AI and Education with Andrew Katz

Travis Williams (00:15.374)

That's really cool. All right And so you're also the director of the ideas lab

Andrew Katz (00:23.833)

I am, I am, yeah. So that's the name. Yeah, yeah, it does have two E's, yeah. Being a little liberal.

Travid Williams (00:25.442)

Am I saying that correctly? Because it's got two E's. And so that's the improving decisions in engineering, education, agents lab.

Andrew Katz (00:35.909)

That is the acronym, yeah. So we loosely speaking focus on decision making within engineering education. And that kind of runs the gamut from students, how students make decisions, how our faculty make decisions, all the way up to like how our students who then transition into their professional careers, so how do engineers actually make decisions as they're working in their professional capacity.

Travis Williams (01:05.444)

That's a pretty new lab. How long has it been around?

Andrew Katz (01:08.573)

So this is I started it. Excuse me. I started it when we got here. So I started in 2019 fall 2019

Andrew Katz (01:18.293)

And I had started a lot of my work previously, it was in the area of engineering ethics and ethics education. And so a lot of that was specifically around ethical decision making. And I thought that I wanted to continue that line of work, but also expand it a little bit more broadly. And I thought that making it beyond just ethical decision making and really decision making in different areas or different arenas as well, it's kind of what fit best.

Travis Williams (01:46.958)

That's really cool. How did you get into education engine? I'm sorry, let me ask you that question again. How did you get into engineering education?

Andrew Katz (01:50.925)

Yeah, sure.

Yeah, that's a good question. I started my undergraduate degree in chemical engineering from Tulane University. I'd gone there in 2005. So Tulane University is in New Orleans, Louisiana. And fall of 2005 is when Hurricane Katrina hit.

I'd gone there to study civil or environmental engineering and then after Katrina, the university closed for that semester and when they reopened for the spring, the university administration had decided to cut five of the seven engineering programs, including civil engineering, which was a little ironic given what happened. And so I decided to stay at Tulane and get an engineering degree I'd been pretty fortunate with scholarships and things like that. So I figured I'll get an engineering degree and then figure out something else as I go along. And so I switched to chemical engineering, which is one of the two that they kept. But it didn't really felt quite like the fit for me. And I knew I wanted to teach, so I knew I wanted to sort of pursue that line and I wasn't sure if I wanted to be teaching at the higher education level or high school. If I wanted to be teaching in higher education, then it seemed like I needed a PhD, so I needed to figure out exactly what was the right path for me. And eventually, I decided to hit pause on the research to make sure I actually did enjoy teaching. So I taught high school physics for a couple of years and that confirmed for me that I was interested in at least teaching. And then I started reflecting.

being more on my own experience as an engineering student and thought that there was probably a path for combining the interest and sort of passion I had for engineering and the passion for education.

And one thing, so personally I'd come from a sort of more like faith-based education system and one of the things that always sort of resonated with me was the idea of developing like a well-rounded person, not just someone who can like work on certain technical problems. And so I was interested in, and that's how I got interested in, ethics education. I was thinking, well, you know, maybe there's parts of how we...educate or train engineers that have room for improvement.

That's kind of how I got into engineering education, and specifically ethics education, within engineering education. And then, you know, maybe to sort of round out the answer. The big thing for me, and this gets back to the beginning of how I started the answer, was like Katrina was like a very obvious example to me of how, you know, decisions that engineers make can have very large impacts on broader populations. And so, I always thought that there are probably better or worse ways to be educating engineers. to be thinking about the impacts of their work on an array of stakeholders and I thought that there was probably some improvements that I could help contribute to in some way.

Travis Williams (05:08.566)

Yeah, that's a really incredible journey that you've been on. I mean, really, to have been in New Orleans at Hurricane Katrina, that's got to change. I mean, I don't imagine you could be there and not be changed in some way.

Andrew Katz (05:26.765)

Yeah, I think that's right.

Travis Williams (05:27.27)

That's really fascinating. So in the area that you're working in, I know that you're doing some work as far as exploring artificial intelligence and maybe some of these language learning, new language learning models related to how we, related to education engineering. Sorry, I keep screwing that up. I'm gonna just re-ask that entire question. In this, yeah, no.

Andrew Katz (05:47.465)

Yeah, no, we're fine. Sure thing. Oh, also time out as you're asking, sorry. Because I know like, because there's like machine learning and there's large language models, I just want to make sure like when you ask the question that it comes out the way you want it to sound. So like large language models is like usually what LLM stands for. So just in case like when you're asking that part.

Travis Williams (06:09.29)

Yeah, so that's a good point. Part of this and me exploring these things is that I don't know what these things are. Like when we titled this podcast, Curious Conversations, they're honestly because like, I'm just really curious and I don't know. I have zero expertise. So thank you for that. So I'm curious, what are you working on related to engineering education in the space of artificial intelligence?

Andrew Katz (06:16.441)

Oh sure. Yeah, yeah, right.

Andrew Katz (06:50.913)

I became interested in this area of machine learning called natural language processing. So natural languages are languages that have evolved through human use like English, Mandarin, Spanish, these kinds of languages. And that's in contrast with formal languages. Like computer languages would be more examples of formal languages. So natural language processing, you know, just trying to get a computer to be able to process the natural language. You should be able to do something with that natural language.

in that because I was interested in how one might scale up qualitative analysis to be able to analyze more data across a broader number of participants in research studies and or in education settings look at how students are responding across like a larger number of students in order to be able in order to maybe be able to make statements that you might not be able to make if you're just looking at like a handful of people.

So I became interested in that and that was back in 2018. So there was like, you know, some good progress. I mean, there's plenty of good progress in that space already. But the reason I gave you that backstory is because I think that's positioned me well for like, basically what's been happening over the last couple of years, which is for these generative models to actually start demonstrating capacities or abilities and what they can do such that they become more useful in lots of contexts.

So what I do is spend a lot of time thinking about how to use those models to either support teaching and or research. So yeah.

Travis Williams (08:34.89)

Okay, when you say models, do you mean stuff like chat GPT? Is that an example of a model?

Andrew Katz (08:39.613)

That is an example of a model. That's a good question. So when I say models, that's everything from these kinds of generative text models. Like you said, chat GBT would be an example of a model that is trained to generate text based off of inputs. That would be an example. But there's other kinds of machine learning models. But I think for what we're talking about, that probably is the closest kind of model that I've spent more time recently doing. I think if we had this conversation a year ago, of models, but now that pretty much is the sort of predominant one, primarily because it can do a lot of things that other more specialized models would do in the past, but it can probably do them as well or better. So if you have a sort of like one...general purpose model that can do well across a range of tasks that kind of saves you from needing to be using specialized models. You know, like, so one model that just analyzes, I don't know, student responses to end of semester feedback. And then another model that, you know, analyzes student teammate comments and stuff like that. Like, no, if you can just have, like, a one general model that can analyze all those, then that's kind of the way you go. So I spent a lot of time thinking about how do you use something like chat.

Although it's not quite what I end up using, but same kind of generative text model. And that's, like I said, so supporting on the research side.

That would be, aside from the examples I just gave, like if you have student responses to exam wrappers. So exam wrappers are like these kinds of reflection essays or questions that students get after they take an exam that are motivated by a bunch of concepts from self-regulated learning. The idea that like, you wanna ask students to reflect on what they've done for their prior preparation so they can be planning forward going into the future for future exams and things like that.

Andrew Katz (10:42.215)

all sort of based in a bunch of theories about metacognition. So you do that, and then if you have a few instructors who maybe have a couple hundred students each, and there's across their sections, you start getting a large number of student writing and student written responses that you want to analyze. And so for me, I'm kind of curious, how do you see different patterns in student self-regulated learning kind of associated with their exam performance? So that's being able to answer those kinds of questions. You know, in the teammate feedback example that I gave earlier, it's kind of looking for instances of potential biases and how students give each other feedback. So do you see different kinds of comments given to other teammates based on or associated with, you know, person, like certain immutable characteristics? Are they like an international student? Does a, you know, a student from one group give, you know...

more negative kinds of comments to certain students from other groups, that kind of thing. So that's all kind of like on the research side. And you know, and that again, the examples I gave were sort of like more shorter writing, but then there's also like analyzing student essays. So it sort of runs the gamut there. And then on the teaching side, Right now it's mostly like how do you help either help students to leverage these models and how do students think about this or how do faculty members kind of address or rather not address but maybe alter their assessment strategies, you know, in response to some of these models because a lot of these models now or at least some of these models now can achieve near human level or at human level performance if you give them a regular homework assignment that you might give a student. So the idea is if you're teaching a class then you may want to modify your assessment strategy to account for that possibility.

Andrew Katz (12:47.237)

And so it's kind of looking at how do faculty members actually approach that. And that gets back to that decision making piece that we were talking about earlier for my lab because it's all about how do faculty members actually make decisions, instructional decisions in their classroom.

Travis Williams (13:02.098)

Yeah, so it sounds like, if I'm understanding you correctly, a lot of the practical applications right now are in maybe combing through large amounts of data, whether they be feedback that people are giving you or maybe assignments and trying to pick up patterns and stuff like that. Is that accurate?

Andrew Katz (13:19.109)

Yeah, that's definitely the sort of like, at least for my focus, that's kind of where I spent a lot of time thinking. There's other kinds of use cases where there are good examples of faculty members using these kinds of models to support their teaching. It's like one thing that I did in the spring was to...C, help students explore how they can use these kinds of models almost as like intelligent tutoring systems. So some kind of system that can help explain concepts to them in ways that are accessible. One activity that we would do would be to ask or prompt the model to explain a concept as if they were in kindergarten, as if they were in high school, as if they were a graduate student, and then kind of look at the differences and the different levels of explanation to identify where there were similarities and where there were differences. And that way it's kind of the idea of, well, there's like a baseline concept that's going to show up in all of those explanations. But then as the model kind of assumes that you are at different levels of your education, then it'll add on levels of complexity.

Andrew Katz (14:28.947)

that extra complexity starts to come in. So that's those kinds of, that's like a more practical, like in class kind of application, I think.

Travis Williams (14:38.799)

Is there any particular model that you found to be more successful than others?

Andrew Katz (14:45.585)

Yeah, that's a good question. So I think at the time we were using chat GBT because that was the only one that was widely accessible, even though even though back in the spring it was still one of those scenarios where you'd get the message that like models that are, you know, the web interfaces at capacity now. You no longer get those kinds of messages, I think. But now there are also other models that are available. So BARD or Claude. So BARD being Google's model and BARD being Anthropics model. I think those are two other examples. And if anything, I spend probably more time using Claude.

for average use cases. And that seems to be pretty good. So it's really more like chachiapetit and clon.

Travis Williams (15:41.95)

Is there anything that you use these models for, like, just personally? Like, does it help you make a grocery list or is there, like...

Andrew Katz (15:47.913)

That's a good question. I will use, oh yeah, so actually I'll use it a lot for help with coding. If I'm programming things. So there's GitHub Copilot, which is a code generation model. It's kind of trained specifically for generating computer programming languages or computer programs. And...

I will use that through an IDE, so an integrated development environment, I'll use it for that to help with coding. But then sometimes if it's a more extensive kind of project that I'm working on that's going to require lots of lines of code or things like that, and I don't think that I would be able to efficiently code that up myself, then I'll definitely just go to ChatGBT, usually in that case actually, rather than Clawed, I'll go to ChatGBT and give it the setup.

trying to accomplish and then prompt it with, you know, help me either complete this code or let's work through this step by step or things like that. And similarly, when I'm troubleshooting, I think it's also been pretty helpful when troubleshooting problems with codes and stuff like that. This actually is a use case that I'll use, maybe not quite a day-to-day, sort of like outside of work, but it's pretty useful.

Travis Williams (17:12.97)

Yeah, that's cool. I told somebody a while back that I had spent a while on Chat GPT, just quizzing it on different movie lines, but it was wrong a lot. And so it made me feel good about myself. But other than that, I haven't played around with it too much, but I know, I think I have heard quite a bit that from people that do more stuff like coding, that maybe it's a little bit more, like that's a little bit more.

Andrew Katz (17:20.509)

Hahaha

Yeah

Travis Williams (17:41.166)

I don't want the right word is I don't want to say accurate, but accurate might be the best word. Then something like I might do, which is like, Hey, give me the best quote from caddyshack and it quotes something from Ghostbusters, uh, that may not be its forte.

Andrew Katz (17:51.717)

Yeah, that's interesting. Yeah, that may not be as forte. You know, the other thing that's tricky about these models though, is that there's a bit of a moving goalpost element to these in the sense that these companies are regularly updating the models and releasing those updated models as they continue to train them. So something that may or may not have worked six months ago may be different now. So I'd be kind of curious to know if you went back and asked it some of those similar kinds of questions. What kind of responses you got?

Travis Williams (18:23.822)

Well, that's true. I've also heard that they, you know, they build off how people interact with them. So the more you enter, the more I ask it quotes from Piaty Shack and give it back stuff, the better, the more it learns them. So.

Andrew Katz (18:36.569)

Yeah, that's a good example too. Another thing that you'll see is people like prompting it for jokes, just like asking it to tell them jokes. That's like another use case, or not use case, but sort of like test that people will give it to see how it's performed.

Travis Williams (18:50.146)

That's a pretty good one. Well, what do you think the potential is for these types of models in education?

Andrew Katz (18:56.841)

Yeah, so I tend to be a little more Optimistic than most well optimistic in the sense of like from the student side for helping them Helping students learn so are you familiar with? Khan Academy So Khan Academy, I'm not going to do a justice and that explanation but Khan Academy is an online platform that instructional materials and resources for a whole range of topics and subjects, everything from like elementary school up through I think intro undergraduate courses. And that platform has been around for a while now. But one thing that they rolled out in the spring when GPT-4 was released was their version of an intelligent tutoring system that had GPT-4 integrated into it. And that's the kind of thing that I think has a very large potential for making a big difference on the education side in the sense that...

There's a world where these kinds of models help give students feedback much quicker on their assignments. And I think closing that feedback loop to speed that up is a big thing. I think helping students, there's this idea from like called the two sigma problem about how...from like the 80s I think. But the idea is that you know if you can help if you can have like a one-on-one instructor for each student then you can kind of shift their performance space like two segments like two standard deviations above kind of like where they are but the problem is that you can't really have at least back then was that you couldn't really have an individual instructor for each student and it is well now with these kinds of models maybe actually can get closer to that kind of scenario. So

So with Khan Academy and they call it Khan MeeGo, is the tutoring system that they have that has GBT-4 integrated into it. The idea for where they're going with that is to have the model babies basically understand what resonates with each student. So for example, maybe you...

Andrew Katz (21:18.001)

I don't know, maybe you like to play soccer, or you grew up playing soccer, or stuff like that. And if that's the case, and I'm trying to explain a concept to you, then maybe it would be great if I could help put things or explain concepts in ways that draw on similar kinds of ideas that you might have from soccer. Or maybe you like playing musical instruments, and same idea. Maybe I can use some sort of metaphors that draw on ideas from music.

Travis Williams (21:40.578)

So it learns who you are and tries to scaffold the information, new information on the pre-existing information, just like a teacher would. Well, that sounds fascinating. And also the idea that you could get that feedback the minute you have that question at 2 a.m. when you're trying to cram for an exam.

Andrew Katz (21:46.089)

Exactly, that's the kind of idea. Exactly. Yep, that's the idea.

Andrew Katz (21:59.157)

Exactly, that's exactly it. That's sort of like 24-7 kind of access that just realistically you would never expect from a typical person. Yeah, sure, sure. So one of the big challenges currently is just the cost of serving these kinds of models. So the cost of making these kinds of models accessible to everyone.

Travis Williams (22:06.43)

Yeah, that's awesome. Well, what are some of the challenges you think standing in the way of us reaching our potential with artificial intelligence?

Andrew Katz (22:28.469)

it's pretty high in terms of both like the hardware, so the kinds of GPUs, so graphical processing units, the kinds of computer hardware that these models run on, they're not the easiest to get your hands on. So that's one thing.

And then second thing, the idea that like, well, right now, it's usually a small number of companies that have the resources to be able to train and make these models available. And they do a fairly OK job making them available and accessible. I don't know if I'd count on that always being the case. In many cases, they are companies that have a bottom line.

And so I think as soon as you start paywalling some of these things, that becomes an issue again of access. So I think there's that part of it. Then there's also the part of it where like...these models, there's no guarantee that these models are like generating factually correct information. Right? So like, there is a scenario where the model is trying to explain something to you, a concept to you, in terms that you might understand, and it just completely gets it wrong. Right? And in that case, like that, that's obviously not very helpful. So there's that kind of thing, I think. Those in my head are kind of two of the bigger. issues in terms of like accurate and correct information and then also just the accessibility.

Travis Williams (24:02.478)

Great, great. Well, to kind of round this whole conversation out, what is something that gives you hope in this space?

Andrew Katz (24:10.621)

That's good. Something gives me hope in this space is just the potential staying on this idea of like the intelligent tutoring systems and the sort of individualized education. I think you know if you can take even a step in that direction then that's a very big improvement. You know, if you spend a lot of time, if you spend any time watching instructional videos on YouTube, I don't know, about any specific topics, maybe a lecture about something or anything else, you will see tons of comments from students, well, presumably from students, talking about how, you know, this video is so great, it helps me understand something better than my professor or something like that. And how accurate all those are is maybe not the point, but the idea is, I like, well, a lot of students who just aren't quite getting what they need out of their current education experience. like their current formal education experiences, so they go sort of pursue other avenues. And so if you have some other sort of ways of helping students access materials and get them that feedback and that sort of interaction that really does lead to better learning than just statically watching a video, I think that gives me a lot of hope and I think there's a lot of potential there. Because it's one thing, you know, to go watch, I don't know, like a 45-minute lecture on something that's been uploaded to YouTube, and sure, I imagine you probably get some somewhat understanding of that topic beyond what you had going into watching that video. But if you can then sort of like be quizzed or have questions asked of you to see how much did you actually understand it, maybe actually be engaged in a dialogue that is based off of whatever it is you just watched. I think that really sort of helps move the needle much more. So it's that kind of thing that really gives me a lot of hope.