Episode 2: Quintin Cutts on modelling Code Comprehension
It’s all very well getting an AI to write your code for you but neither writing code or reading code are the same as understanding code. So what is going on in novices brains when they learn to actually understand the code they are reading and writing? In episode 2 of our community podcast, we spoke to Quintin Cutts from the University of Glasgow about his paper co-authored with Maria Kallia on Introducing Modelling and Code Comprehension from the First Days of an Introductory Programming Class in published in CEP ’23: Proceedings of 7th Conference on Computing Education Practice, from the abstract:
An approach to code comprehension in an introductory programming class is presented, drawing on the Text Surface, Functional and Machine aspects of Schulte’s Block Model, and emphasising programming as a modelling activity involving problem and machine domains. To visually connect the domains and a program, a key diagram conceptualising the three aspects lies at the approach’s heart, alongside instructional exposition and exercises, which are all presented. Students find the approach challenging initially, but most recognise its value later, and identify, unexpectedly, the value of the approach for problem decomposition, planning and coding.
Full transript and selected references below, cite this blog post using DOI:10.59350/dvcb8-gvd18
References
- Listen to this episode online at pod.co/the-rest-is-teaching/quintin-cutts
- Subscribe and listen wherever you get your podcasts, details at uki-sigcse.acm.org/podcast
- Quintin Cutts and Maria Kallia (2023) Introducing Modelling and Code Comprehension from the First Days of an Introductory Programming Class in CEP ’23: Proceedings of 7th Conference on Computing Education Practice Pages 21–24 DOI:10.1145/3573260.3573266
- This paper was discussed at ACM SIGCSE journal club meeting № 33 see DOI:10.59350/sigcse.1698
Episode transcript, machine generated
⚠️ DISCLAIMER: Please note this transcript has been generated using speech-to-text software and contains transcription errors and speech disfluency. ⚠️
Duncan: [00:00:00] Hello and welcome to the Rest is Teaching podcast for computing education researchers and practitioners. In this podcast, we’ll meet people who are changing the way that we teach computing from school through to university and beyond. What is computer science anyway? Why should people learn it and how can we improve the way that it’s taught?
My name is Duncan. I’m your host and we’ll be meeting authors of studies and papers that tackle these important issues from our journal club. What is their practice and research? Why is it important, and how can their insights be useful to other people teaching computing in any area of education and at any level?
My guest today is Quintin Cutts. We talked about his paper co-authored with Maria Kallia in 2023 on introducing modelling and code comprehension from the first days of an introductory programming class that was published in CEP: the Computing Education Practice Conference at the University of Durham.[00:01:00]
Welcome Quintin. Thank you for joining us. Thank well, thank you for coming to the, to the Journal Club and, and presenting your paper. It was a really good discussion, but before we dig into the paper, let’s, can we start with some biographical stuff? So can you say for people who don’t know you, I mean I know who you are, but for people who don’t know you, can you say who you are and what you do?
Quintin: I’m Quintin Cutts and I’m a professor of Computer Science Education at the University of Glasgow. Um, I’ve been involved in school education, university education, liaising with industry curriculum development, researching, how we learn this stuff. Lots of things.
Duncan: Good stuff. So now going back a bit further in time, I’ve started asking students questions about why they studied computer science, and I found that the answers have been fascinating because everyone gives a completely different [00:02:00] answer.
So I presume you studied computer science as an undergraduate in some shape. Sure. Yes. So, um, can you remember what got you into computing in the first place?
Quintin: A Commodore PET. Arrived in my school when I was about 13 or 14. Ah. Um, and my secondary school has now become a sixth form college, and it was in Colchester in Essex.
And I, I just happened to be googling around a driving route, um, recently. And, and, and I went through culture and I looked and there was my school. And amazingly there was this huge building called computer science in and, and in fact that that building was, you know, only about. I dunno, 50 yards from where that Commodore Pet was, even though the buildings that the Commodore pet was in, all knocked down.
It just took me back down memory lane. And I, I remember thinking at the time that Commodore Pet was there, you know? Bill Gates was, you know, doing the very first versions of DOS and all that kind of stuff, you know, so, [00:03:00] yeah, so it was Commodore PET that got me into it. And then, and then I went down the Sinclair ZX spectrum route and then, um, that was enough really.
Duncan: Okay, good. That’s good to hear. Um. And then, so then the other, the other question here then is, so why computing education research? ’cause obviously, you know, if you’re gonna go into academia, going into research, there’s lots of things that you could choose to do research on. What was it about computing education, research that, um, um, that got, you know, got you into it?
Quintin: Yeah, I, um, I was lucky enough early in my lecturing career, uh, I mean, I’ve done quite a lot of educational stuff. I mean, teaching people things. I quite obviously quite enjoyed doing that. So that was the education part. Um, but education, research, I was lucky enough to fall into a group of. Five academics and um, a kind of an academic mentor kind of person.
Um, and we were each in different departments. This was about the third, second or third year I was, I was here at Glasgow [00:04:00] and uh, we all had our own individual project, but we all came together and talked about academia and what a nightmare it was and that kind of thing. Um, and the question I had right back then, and that is 20, 25 years ago, um.
The question back then was, can anybody learn to program? Um, and that, that was in a way, that was my first sort of research question, but actually that’s been a thread through my entire career in different ways to answer that question and then to understand what it might take to enable anybody to learn to program.
So it’s, it’s been a nice thread.
Duncan: Yeah. Okay. Good. And then, um, so before we dig into the paper, so the paper’s about, um, code comprehension or what’s sometimes called. That’s the same thing as program comprehension, I think, isn’t it? So can you say, I mean that falls on from what you’ve just said, there actually is, anyone can learn to program potentially.
Mm-hmm. Um, what is code comprehension and [00:05:00] why? Why does it matter?
Quintin: I suppose when, when one says, can anyone learn to program, what does it mean to be able to program? And, um, uh, I think it’s, it’s, you know, even to say I’m going to program almost like means I’m going to go and create some stuff. I’m going to go and write some code and.
Certainly when I was growing up, university courses seemed to be very, very much based around writing code. Um, and not very much attention was paid to. Could I understand what I’d written? Could I understand what somebody else had written? Um, and so, so code comprehension, program comprehension is. The exercise of understanding what’s written on paper and, sorry, what, what, you know, what I’ve got in front of me, not on paper.
Sorry. What’s, what’s, what’s on the screen in front of me? Can, can I understand the code I’ve written or can I understand the code somebody else has written? And, and, and I guess it, I guess it matters because, you know, how can you hope that a student is going to have any [00:06:00] mastery over something they’ve written if they can’t understand it?
Um, and how much help do we put. Do we give them to actually enable ’em to understand what’s in front of them and how, you know, it’s, where’s the balance between understanding what they’ve created and just simply getting something over the line and, and giving a result, you know, a, a working program. So, so very often it, it might appear to a student that what we want ’em to do is create a working program.
It may not be so obvious that what we really want ’em to do is to understand that working program.
Duncan: Yeah. Yeah, that’s interesting. Sorry. There was something in the, in the May issue of communications, the ACM, someone tried to con quantify how much time your average software engineer spends doing code comprehension and they put the figure at between 60 and 70% of a, I mean, it’s kind of perhaps obvious when you think about it, but nice to have a number.
70% of the software engineer’s time is spent trying to understand either their own code or more usually somebody [00:07:00] else’s code. So even when you’re a pro, you spend a lot of time doing this. It’s not just something that beginners do.
Quintin: No, no, I think it’s, I think it, I think another reason it’s interesting and I’m, I’m only just uncovering this.
Yeah, I mean, I mean obviously we want somebody learning to program to be able to understand what it is they’re creating. Obviously there are many ways in which an industry person needs to be able to understand code. ’cause they might be doing code review of somebody else’s code, or they might be looking stuff up on the web, or they might be picking up a huge new system and they’ve got a huge code base and they’ve got to understand it.
So there’s, there’s all those kind of reasons. There was another reason that came to me. I dunno, a couple of summers ago when I had a master’s student, um, and she came in and, and I, I just said, well, look, you know, you are, you are in the computing education context. You know, what’s been challenging to you as you’ve been trying to learn this?
And she was a transfer student, so some other subject in learning computing. So she was effectively like learning computing for the first time sort of thing. And, and she really identified the challenge of code going up on a [00:08:00] overhead slide, you know, in a lecture. She was obviously expected to understand in real time in the lecture now, and not the introductory course, but then in, say in an algorithms course.
And she was expected to understand this. There was just no way. So she’d developed all sorts of clever techniques for understanding the code after the lecture, and she could tell that her friends hadn’t. And so, you know, but, but she realized she needed to understand anyway, so, so, so my point is, I mean, isn’t that fascinating how many of us teach later computer science courses?
Put bits of code up and expect our students to be able to understand them because we could, but they, you know, but they may not have developed that skill yet. So it’s yet another reason why we might need code comprehension is because later courses in computer science depend on being able to read code and understand it.
Duncan: Yeah, yeah, definitely. Okay. So that the, the title of the paper that we discussed that you wrote is called Introducing Modeling and Code Comprehension from the First Days of an Interruptive Programming class. So. Um, in a nutshell. [00:09:00] So what, what is the paper about? At the beginning we talked about what, what is it about and where does it fit?
And I think we’ve probably already covered some of the why it’s important. So what is it, what is the paper about in a nutshell?
Quintin: Um, it introduces an approach. I mean, it was obviously in the computing education practice conference. So it is a piece of practice. It, it, it’s an approach I’ve used, um, really only a couple of times and a couple of runs of the course, but to encourage students to see the complexity of understanding code from really early on.
Um, and I suppose the point there is that, um. When you show students a program, it may well look to them like a piece of text. Well, it doesn’t look like a piece of text. Yeah. I mean it is textual and there it is sitting in front of them and they’ve had a lifetime of understanding how to look at pieces of text in natural languages.
Um, and. [00:10:00] Code comprehension is really more complicated in a sense than natural language comprehension because we, you know, I, I look at a piece of English in front of me and I’m not really any more thinking about the grammar and the way the, the, the way the language works. I’m thinking about the, the meaning up in the real world.
What’s this, what’s this piece of English text describing up in the real world? Um, sorry, I’m not really summarizing this, but, but so, so, so, so that’s, that’s, um, that’s what we do, um, in our natural language. We, we, as it were, if you think of the text here, we are just trying to work out what that means in terms of the real world that it’s describing, the real world situations.
Now, with, with, with, you know, what this, what this, um. Teaching method really tries to get across right from the very earliest point is that when you understand code, you’ve got to understand it in three different ways. So you’ve got to understand the grammatical structure, uh, that, that’s my sort of level, that’s like the text or, you know, in this, [00:11:00] this follows, uh, schulte’s block model.
So it it’s the, the tech surface. You’ve got to understand the tech surface, but then, then you need to understand how, what’s written there. How do pieces of that text, the grammatical pieces, how do they map up to the problem that’s being solved, um, and how do they map down to the kind of semantics or to program execution?
Um, and, and there’s, there’s the, there, there’s very much, those kind of three, three levels. Um.
Duncan: The di, the di the diagrams in the paper are nice ’cause they explain it. It’s quite difficult to explain the, yeah,
Quintin: I’m, I’m just sort of going up and down like this, but, but sort of three parts I think is important, but to see, um, you know, codes in the middle.
Um, and one way of thinking about the code is that it, it’s connected, you know, things like variable names and comments. They all connect the program up to the problem it’s trying to solve. ’cause you try and use meaningful, you know, when you say meaningful variable names, they’re meaningful because they [00:12:00] describe, they connect us to the problem.
Um, but. As we all know, you could take, you could use any old variable names you like, and the same mechanism would be created downwards, the same executing program. It would be exactly the same. So we need to understand that whole mechanism and that, that kind of brings out the whole idea that, you know, programming is all about modeling.
You know, I’m thinking of something in my problem domain, and I’m modeling it using this set of tools down in, in, in my language domain.
Duncan: Okay. So that, that, now that some, that answers all those three questions in one go. I think so. Um, I suppose one sort other thing that you sort of mentioned in the paper, and you sort of alluded to there, was that, um, reading code is different to reading English language.
So you’ve got the, I suppose you’ve got the syntax and the semantics when you are reading natural language, but you’ve got this, is it this third, um, this third dimension that’s different when we’re reading code? Or is it different on all three levels?
Quintin: Um. Yes. Uh, I’m going [00:13:00] to display, uh, I’m going to display our fear, a level of ignorance about natural languages.
But, uh, uh, I just sort of think that when, when we look at a natural language, it’s just like we kind of, we read it and it turns into, it turns into a meaning. I mean, so we’ve got the syntax there, and as you say, the semantics I would take as being the interpretation of those things in the real world that.
The picture they create, as you might say.
Duncan: Yeah. I mean, you, so Jane Wait’s always keen on saying, you know, think, think about how much time in primary and secondary school people spend on a natural language. Like, oh God. Yeah. And then think about how much time you spend on, you know, comprehension in, in code.
And it’s like you just, just skipped over very, very quickly. And lots of assumptions are made as to whether people are gonna understand it or not.
Quintin: Absolutely. I mean, that was an exercise that I used with teachers, um, a long way back, you know, to, to show them a piece of natural language and say, tell me everything about this piece of natural [00:14:00] language.
You can. And they would tell me at all different levels and all kinds of stuff, you know, and say, how long do you give your students to, how long would it take to learn that? And they go about six or seven years before they can understand that, you know? And then yeah, yeah,
Duncan: yeah.
Quintin: Show them visual basic up here, you know, tell me all that.
And then how long were that two weeks, you know? And, um.
Duncan: So, um, we, you mentioned this block model. Now I, I, I, I guess a lot of people would be familiar with it, but for those that are not, what is the block model and why is it interesting and important in this particular context? Yes.
Quintin: Uh, block model was developed by, uh, Carson Schulte in about 2008, round about that time.
Um, and it’s a way of categorizing the different dimensions of, uh, code comprehension and so. Uh, he has it as, um, a series of, it’s, it’s like a, a grid of squares, three by four, three wide, four tool. Um, and each one of those squares is known as a block. So that’s why, that’s one reason why it’s called the block model.[00:15:00]
Um, and the three columns relate to the three pieces I was talking about, tech surface. And then the, the meaning of the program in terms of the problem, he calls that the functional side kind of thing. So if I go upwards, I’m talking about the, the, the function of the program in terms of the problem it’s trying to solve.
Um, and then the other side is the execution side, um, which like the program execution, what’s actually happening in the, the semantics of the programming language. Uh, sort of a bit like the, the notional machine or the virtual machine created by the programming language. So he has those three columns, text, text, surface, machine side, functional side.
Um, and so obviously you need to understand programs on those three dimensions, but then he is got four. Um, as it were, levels of complexity. So the bottom one is atoms. So those are the most primitive parts of the programming language, um, that you need to understand. You, I need to understand the, [00:16:00] the syntax of expressions or something, um, or, or a variable.
And so you, you’ve got the atom. You’ve then got blocks, which are slightly awkward, that, that you’ve got this block level and it’s called a block model, but this block is slightly different. So that’s when you’ve got, you know, a few lines together. But the, the code comprehension part now is as an entity.
What do those few lines do? Just as one piece. So don’t think of it now as four lines and a bunch of atoms. Think of it as one piece. And then you’ve got the relational level, which is when you’ve got piece of code, pieces of code that are not contiguous, you know, lines together, but, but might be around different parts of the program.
So the relationships between different parts, um, that might be modules or using parameters or a counter variable with a declaration, a test, and an update. Um, and then you finally got macro, which is explaining the whole program. So, and,
Duncan: and this is, this is a theory. I mean, is it had to call a fair to call a theory?
I suppose it’s a [00:17:00] kind of useful or a framework that, I mean, I think it’s useful, but it’s used all over the place in lots of other, in other kind of research as well, isn’t it?
Quintin: Uh uh, yeah. I just find it a very, very useful framework to help with, um, um, understanding. What we’re trying to do when, well, what we’re trying to do when we understand programs, but, but it, it, it’s not entirely unrelated to writing programs as well.
In terms of, you know, what, what students said about, about this, this model we put across to them about, you know, the functional side, the, the tech surface and the machine side. Um. I was always, yeah, I was thinking of that as only really useful in terms of reading code, but they actually said no. It was actually really helpful in terms of thinking about problem solving.
’cause it gave them a clearer idea of what they were actually trying to do. I think it really brought the idea of modeling out to them, you know, that that, that there is this thing I’m creating using the tools of a programming language, but. It’s got to match to the problem I’m trying to solve kind of [00:18:00] thing.
And, and yeah. So somehow it was helpful.
Duncan: The, the, the model thing is interesting. So not, not talking about the block model, but the, you’re talking about, um, uh, so um, computing as a modeling activity. Yeah. So what is it that students are, so they’re not, they’re not just programming, they’re modeling something.
What is it students are modeling when. Writing code in an introductory programming class? Is it the problem or solution or, um,
Quintin: there’s a, there’s a problem and, uh, a problem in the real world or in some context there’s a problem in a, in a, in a domain, you know? Um, so I mean, you know, one domain you might imagine is, is typical one in an introductory course would be, you know, students and exam marks or something like that, you know, so, so there’s a, there’s a problem domain they have to think about and somehow.
They, they’ve then whatever task has been asked about in that domain. So perhaps the task is to find the largest exam score or find the average exam score. [00:19:00] That’d be a typical one, wouldn’t it? So then, okay, well that’s great. Now I’ve got my set of tools rather, rather like my Lego toolbox of blocks, you know, how am I going to model this task using this set of tools?
Um, and so that’s the modeling activity because, um, you know. Um, there are many different ways I could go about that particular task. Yeah.
Duncan: Yeah. Okay,
Quintin: so, um, so, um, let, lemme just say a bit about that. I mean, I think it’s gone. Yeah, I, I, I think, I think that understanding of programming is modeling. I think people, uh, I think people miss it sometimes.
They quite understand the ramifications of it. You know, you can’t, you can’t be a good modeler if you don’t understand the problem domain really well. Whatever that problem domain is. And, and the solution domain, either the programming language and all of that. And, um. I, I, um, I once set an exam question about the [00:20:00] UK number plate system, you know, the way, you’ve got two letters, two digits, and then three letters.
And, uh, the, the question was all about working out where the car had been registered and which six month period it had been registered in. ’cause that’s the way new and home based system worked. And I suddenly thought, my god, that’s a really unfair question. ’cause even if I thoroughly explain how that system works in the exam, some students are gonna have to spend ages working out what it all means if they don’t come from this country or they’ve never bothered to understand.
Whereas other students have a huge advantage ’cause immediately they know everything about it. I, they understand the problem domain, the whole area. Then there’s a particular problem within it to actually solve. And actually in that case, I thought, well, I think I, I’m gonna have to tell ’em about this. You know, so, so I, I then, you know, before the exam, I spent time in the lecture saying, you are gonna get a question about the UK number plate system.
You need to understand about it. Let me explain it fully how it works, so that they’d all come in prepared to the exam. And then that felt a lot fairer. And then I was, and, and how often do you think we give students problems that either. [00:21:00] They, they know nothing about. So it’s unfair. Or, or, you know, they’re, they’re really boring.
I mean, yeah. Paying attention to the problem domain. Perhaps we should do a little bit more of that. Um,
Duncan: good. Alright, so, um, we had a few questions from people in, uh, in the journal club meeting. So this was from Paul in Lancaster was asking about. How do you separate the functional and the machine side for, for sort of simpler programs where they’re not just the same, you know, rephrasing the same thing.
You know, with a complex program, there’s gonna be a big difference between the functional side and the machine side. How do you go about separating that for much simpler problems? Or is that, is that an issue? How much is that an issue? I,
Quintin: I think it, I think it is something that the. Um, these, these kind of questions come up in, in tutorial exercises for the students once this idea has been introduced in the lectures.
And, uh, I think it is [00:22:00] something to play with and to explore and it, it does seem to take the students a while to get the difference. But, um, you know, I, I can describe a, um, I can describe a variable declaration. In a machine side by simply saying, well, what this line does is it, is it creates, um, a variable called whatever the name is, and it, it, it, it sets it the value to zero.
Uh, say, say it was a counter. Yeah. But, but then I could also say. On the functional side, I could say. Right. Well, this line is to set up the counter that we need in this problem because we’ve been asked to work out the average and we need to know how many items there are. So, so this is to, this is to create the thing that’s gonna remember how many, how many items.
So, so you can see that second definition is entirely in relation to the problem. Yeah. Whereas, whereas on the, you know, if I look at our side and the function side, it’s creating a variable called count, setting it to zero. Yeah. It doesn’t tell me anything about a problem. [00:23:00]
Duncan: Yeah,
Quintin: yeah, yeah. Okay. So, so, so you can, and it, it might seem a bit labored to start with, but it, it, you know, and in fact when I was, when I explained the very first program the students see, which is a simple input process, output problem that we’ve really thoroughly understood in, in human terms, I, in, in, in the problem domain side.
Yeah. It, it’s quite laborious. And the amazing thing is even the simplest program is really complicated and we get scared. About showing how complicated it is, and we do our students a disservice by doing that because we sort of leave it up to the, you know, we, we leave it great. We say, oh, don’t worry about that.
Yeah. Well, no,
Duncan: it’s the teach yourself programming in seven days probably, isn’t it? Kind of, yeah. It’s really easy. You know, anyone can do this. Oh, hang on a minute.
Quintin: But perhaps if you, but perhaps if we got the, the absolute basics in place, the fundamentals in place, everything else would be much easier.
Duncan: Yeah. Yeah. Okay. Right. So, um, [00:24:00] there was a question from Troy about what do you use to explain the machine understanding? I think this was sort of touching a little bit on, you’ve already talked about notional machines, so how do you explain machine, the machine understanding? ’cause that’s one of your three dimensions of Yeah, yeah, yeah.
Definitely. How you explain what machine understanding is to somebody who’s only just started programming.
Quintin: Yeah. Okay. So the first, uh, I do in the paper I do use the phrase machine understanding, and I think perhaps the word machine is, is a mistake. Um, just because people start thinking, oh my God, he’s talking about the bits and bites and registers and so yeah, they really think machine and I, I, I don’t mean that at all.
I’m, I’m really thinking of it in terms of Benji Bole definition of notional machine, which was kind of a, a minimal model. That can be used to describe the computation that’s going on or, or the execution of the program. Um, and then also the kind of almost extension of that to, to his and Sally [00:25:00] Fincher, slightly adjusted definition of notional machine, which is, which is any teaching tool that helps to explain what are otherwise kind of hidden as, uh, uh, um, execution aspects of a program.
So, for example, you know that, that very simple introductory program. Read in a couple of numbers from a file, um, multiplied them together, whatever it did, I can’t remember, and then, and then wrote out the answer. But of course, there’s some hidden stuff in there. You can’t really see the file, you can’t really see what’s happening.
When you read the file, you can’t see the variables. So, so, you know, very early on this was just all on slides and just giving them what, what Sally and Ben would call the notional, you know, notional machines to describe, to give them a representation of a file, you know, as a sort of, as a sort of document with characters in it and new line characters and a position where I’ve got to, when I’m reading to try and make it completely concrete how that read function is gonna work.
Um, yeah. [00:26:00] So, so, um, so very much pencil and paper later on. Um, we would use things like Python tutor to actually show more in a more complex way what’s happening inside the machine. But again, you know, ’cause Python tutor gives you a notional machine representation of objects and stack frames and the code, and the code pointer and all that kind of thing.
So it, it’s very helpful in actually letting the students, um. See in, in a more efficient way than loads of slides.
Duncan: Yeah. So I, the, the, the, the, it could, the machine could be simply, I think they used an example of pencil and paper kind of thing. Yeah. Just be a sketch. Or it could be, you know, it could be quite a detailed, um, mach model of the machine light.
Something like Python tutor, for example, which goes into a lot of detail and talks about.
Quintin: I mean, it could, I mean, so if, if, I mean, if, if, if anybody’s interested in this, when they, when they watch this podcast, they can always drop me a line and I, and, [00:27:00] or you can put this somehow associated with a podcast, but there’s, there, there was a paper that Peter Donaldson and I wrote about a notional machine thing that we’d used in schools that was pencil and paper, which really required a student to explain every single bit of executing a program.
And then their whole understanding was captured on the paper. Then we could look at that and see whether you’re very quick to work out whether they were correct or not. Yeah. So, so that, that’s one. Or I could, or, I mean, I can always send a PowerPoint file that gives the full description one or two slides of which are in this paper.
Duncan: Yeah. Okay. Good. I’ll, we’ll, we’ll put that stuff in the show notes as a sort of text comes with the, with the, um, audio. Um, so there’s a question from Rose in Strath. Clyde was, um. Outcomes. Did you, were you able to sort of measure what the outcomes were? And, and so one of the things you’re saying is that it gave, one of the, one of the nice outcomes was it gives the students this language to [00:28:00] explain to other students when they’re talking about what they’ve done.
Mm-hmm. Um, it gave ’em a language to talk about that is there was, were you able to ob observe any sort of, or record any sort of performance enhancement of saying, well, students who’ve done this. Now I’ve done this kind of explicit code comprehension step and now outperforming students later on who hadn’t done that.
I mean, I guess that’s, that’s quite a difficult thing to do. But was there any Go on.
Quintin: Yeah, I don’t, um.
I didn’t, I didn’t think ahead far enough on whether I was going to be thoroughly evaluating this method. This was, you know, this is de definitely kind of an e uh, evolutionary process to get to get here. Um, um, once I decided to write this paper, ’cause I thought there was really something worth it. You, it really seemed valuable to explain these three levels and blah, blah, blah to the students and, and get ’em [00:29:00] work on it early.
Um, uh. The evaluation in the paper is because we, we wrote out to the class at the end of that academic year, or even the beginning of the next academic year, just to say, look, remember that stuff we did at the beginning? What did you think of it? Um, uh, was you, did you think it was valuable? And, and we had about 10% of the class respond.
Um, and I mean, you know, what Rose says there is, is is one of the thing, get it. They felt they were more able to talk to their peers about code than when they talked to peers who had come through another route and were now in their courses kind of thing. They, they really felt this had given them a way to be more expressive about what, what, what they’d threatened and created.
Um. As I say that it, we did find it right? I mean, you know, some students just said, yeah. ’cause we also said, well, you know, do you think we did it for long enough? And some students said, no, could have [00:30:00] done it a bit more. Others said, no, that was just about right. You know, it, it, it, it got me thinking in the right way about programs kind of thing.
Um, and I think that’s probably the value of doing it right at the beginning. Um. Yeah.
Duncan: Okay. Um, so we had questions from James Davenport and Bath and Alan Harrison about, I suppose it’s sort of when do you stop using it? You know, is it, is it that, um, alluding to that question earlier about, you know, does it break down when you get to much more complicated problems, perhaps, um, when, when do you stop using and say, okay, that’s the sort of scaffolding you’ve learned that now, now we can sort of.
Get rid of that and now you know what you’re
Quintin: doing. Well, yeah, it’s a good question, I think, um,
so I was using it very rigorously for the fir literally the first few programs introduced to the students. And it, and it is quite laborious because, um, you know, you, you, you work your way through [00:31:00] the, the problem thinking about this as a human, how would I go about solving, you really come up with an idealized model of how you’re gonna solve this problem.
Um, and then you show them the code and, and the approach was, you know, sort of, right, well let’s, let’s work out the tech surface and on the slides, put boxes around things and really show the structure so they’re very clear about that. Um, and then go through the, the functional side and then the machine side.
And, and so it, it’s quite laborious. Um, and so I. And, and to back that up then to give them tutorial exercises and say, right, well look, you know, here’s a piece of code to solve this problem. Can you identify, you know, what kind of a thing is on this line? So, I mean, like, you’re basically asking, you are basically testing their ability to give you information back about the tech surface, the, the, the functional side and the machine side.
Um, but I don’t think you’d, this one, these really interesting questions that, you know, on, on the whole, the students seem to think they’d had a, that they’d seen enough, but [00:32:00] some students. Definitely hadn’t got it. I mean, it’s not like this solved everything in my introductory programming class. Um, if only, yeah, if only, but no, this, this was not a universal panacea.
Some, some students who maybe they missed the first few lectures or, um, maybe they just weren’t paying attention or for, or there was something deeper going on, you know, they, it just sort of went over their head and, and later in the course you were aware that they still actually couldn’t really understand their, their code at all, so they needed more of it or something.
So, so it is, I mean. You wouldn’t wanna carry on doing this for the whole course ’cause it was so labor intensive to show them these programs. But yeah. Yeah. I think that doesn’t stop you talking about machine side and function side later on, you know, now you’ve got the language. You can just bring it in when you, when you need to.
Or you say, well you, let’s make absolutely sure we’ve got the tech surface understood about this new construct. And um, kind of take it from there.
Duncan: Yeah. Okay, good. Um, what else have we got here? So, um, [00:33:00] question from Phil Hackett, which is about, uh, um, so were you having to, did you observe any effects with students who perhaps had preconceived ideas?
So students who’ve, who’ve got, um, who’ve done, you know, this is a common problem in first year introductory computer science courses, is that. Some students have no experience and some students already have experience and sometimes that experience gets in the way. ’cause you have to sort of demolish all that preconceived ideas about what they’re doing.
Is that, is, is that something that that helps in terms of, in terms of doing that? Or is was it different? I mean, I,
Quintin: I like that idea of the, the breaking it down piece. I, I remember my own lecturers back in sort of 1984 saying, oh my God, I wish people came in with no programming experience at all, and we could just start again.
Um, I found out, well actually, given the, uh, language transfer work I’ve done with Ethel Trick in more recent years. Um, I mean, how wrong they were, you know? [00:34:00] But anyway, uh, right. So to to, to this particular question, um, uh, I don’t think I can give you a very good answer on that, unfortunately, because my course was for people who had done no programming and the people who had done programming before were on a separate parallel course.
So we have two courses in Glasgow. Yeah. Okay. Um, however, I’m pretty sure that some of those students in the article course needed this understanding and didn’t have it because. People self-select into a more advanced course, but they shouldn’t always be there. Yeah. So, um, I’m just trying to think what would happen.
I think it just depends the kind of, you know, how you get, I mean, I’m being cheeky here, I’m sure, but you know, you get some arrogant students who are convinced they know everything and won’t listen to anything. Um, one of my PhD students for Nula is, you know. It’s very hot on why people find it so difficult to change conceptions, you know, to change understanding.
And it, it’s, it’s much lazier and easier to keep with what you know. Yeah. That’s one reason [00:35:00] why maybe some of these students don’t want to make a change, but some, some students I think would find this really interesting if they’re interested in language and interested in languages and yeah, it depends how deeply interested they are in, in the craft and art of programming because it, it deepens your understanding of what it might well deepen the understanding of what’s going on.
So I’m not convinced it would be a. I’m not quite sure how much knocking down you would need to do for, for, uh, advanced students.
Duncan: Yeah.
Quintin: Okay. Depends on their attitude.
Duncan: Um, so the, so leading on from that, actually, we have questions about does it matter the kind of programming language you’re doing? So this is procedural, uh, type languages.
Is there a, is there a different, you know, does it, does it matter whether you’re using, I mean, so, uh, David was, David Sutter was talking about, uh, Lambda Calculus for example, and, uh, Paul was asking about functional programming languages. Does it, does it work? With all of those as well, or should it, is it, is it, it doesn’t matter what kind of language.
Quintin: [00:36:00] The, the further apart the problem domain is from the, the execution model, the kind of programming environment, the further apart those two are, the, the easier it is to see that there is indeed a machine understanding required and, uh, sort of connection up to the, to the problem domain. I mean, if you using machine code quite clearly, that machine code does not look anything like a normal language or just a description.
But, uh, you know, I’ve always kind of said, you know, one of the challenges with something like Scratch is, scratch is so close, Alice. You know, it’s so close to the problem domain. You know, if I, if I, if I’m trying to move a cow across the screen, I just go, cow, move 10. You know, that language is, is virtually the problem.
Domains language. Actually, it’s the machine domain, so, so it’s actually harder to separate these two domains. It’s still modeling. You know, I’m still, I’ve still got a formal modeling environment in which I’m getting my cow to move. Um, but, but it doesn’t look very different and it doesn’t, it doesn’t sort of seem [00:37:00] very different, you know, compared to adding up a bunch of exam marks and I’m using lists and arrays and variables.
I mean, it, it’s very different, isn’t it? Yeah. So, um, but, but it would apply to all these things. I mean, yeah. I see there in the question mean asking about functional languages or lambda calculus. I mean, you know, they’re all the modeling domain, you know? Yeah. So it still kind of applies.
Duncan: Yeah. Okay, good. Um, it’s just, uh, I think it’s just a couple more questions to get through before we finish, but, um, so we had a question from ihi, um, about, um, hierarchies.
Are these, are these in a hierarchy or are they all at the same level? So, you know, for example, do you need to understand syntax and semantics before you look at machine side? Say, or, um, um, you know, is there, is there a hierarchy, I guess is the question?
Quintin: I think you’re gonna be,
I mean, people end up [00:38:00] writing programs by copying chunks without necessarily understanding the tech surface structure of that chunk, um, or the machine side of the chunk. They just know it does some something that they’re interested in, so they, they kind of know what it does functionally. So they might essentially, they might have a block.
They know functionally what it does. So they think, great, I’ll pop that into my program. Of course, the fact that they don’t, I mean, great. Okay, alright, thanks for your time, necessarily understand what it’s actually doing. And because they may not understand the tech structure well enough, they might not know how to break it down or to adjust it or whatever, you know, they, um, it is gonna cause ’em trouble.
So, um, so
Duncan: I think you call that sort of pattern, sort of pattern matching approach to, to programming where you are kind of. I don’t really know what this bit does, but if I chuck this bit in here, it seems to do what I wanted to
Quintin: Yeah, yeah. That how it’s working. Um, and we all know students do, do that sometimes.
Um, yeah. I don’t, I don’t, [00:39:00] I think, I think you take these things forward in lockstep. I mean, I personally, I think, I think, um, you know, I’d, well, I. To keep it relevant in some way, I would want to introduce new computational constructs because they’re required to solve a particular problem. And so in that case, I’m going to be describing tech, surface machine and, and function All at the same, yeah, all in the same lecture anyway.
I mean, who, who knows what order I’ll do it in, but, um, um, so no, I, I don’t think there’s a really. It’s not like I’d explain all the tech surface first and then all the machine and then all that, you know, I think they, they kind of go in, in, in, uh, lockstep.
Duncan: Perhaps it is a computer science problem to start thinking in things of layers and hierarchies.
’cause that’s how,
Quintin: that’s how people are. No, well, no, absolutely. That’s
Duncan: okay. Good. Um, so, um, what else have we got here? So, um, a few questions to wrap up then. So I think we’ve alluded to this before in [00:40:00] terms of we, we, we, um, any plans to repeat this with perhaps larger groups? That was a question from James Davenport about.
Um,
Quintin: yeah, you
Duncan: know, I mean, because you, you kind of put stuff out there and then actually, you know, maybe, maybe this could be reused and maybe it could be, you know, do some of that evaluation, um, that, that you weren’t able to do or somebody else could
Quintin: do it for you. So this paper was based on teaching in, uh, 2220, no, this is 2, 2 3 in 21, 22.
Uh, I wasn’t teaching in 22, 23, but I will be teaching again this coming year. So. Um, I’ll probably be involved in some way. Uh, I’m teaching jointly with Maria Callier, so I’m not quite sure how we’re gonna organize that.
Duncan: And she, we should explain actually, ’cause we haven’t credited Maria. So Maria’s the, the co-author of this paper, she also did some of the evaluation, um, that on, on this
Quintin: piece.
And you, she was kind of [00:41:00] teaching, well, she drew on my materials just ’cause she took over my course for that year. So, I mean, she’ll have put her own unique stamp on it as well. But, um. Um, I, I think one of the most interesting things that came out, not, not one of the most interesting, but an interesting thing that came out of this exercise, um, was realizing.
Uh, we, we talk about this in the paper was, was realizing, you know, maybe seven or eight weeks into the course when using pythontutor.com to explain some machine side stuff. I mean, this was just completely going over some students’ heads. Like I said, you know, it’s clearly, clearly not getting it. And yeah, and, and that’s what kind of really got me thinking about the idea of students who just read the code as if it’s natural language.
And we kind of call them no machine model students, you know, they, they just don’t. And so. So this is what I kind of mean about if you can get some building blocks in place, then it becomes much easier to accelerate your learning. But if some building blocks are not in [00:42:00] place right at the beginning, you’re completely lost with a student.
So if a student does not understand that there really is a machine side that they need to get a grip of, how can they possibly go forward in this subject? So what we’re interested in doing is, you know, what would it take to identify a student with no machine model, and then what would it do to very quickly help them see the world this way?
To have these kind of three parts. So, so that, that’s definitely a, a, a, a next step for this work. And, and that, that came up with, um, Leo Porter in, in uc, San Diego. He was, he was very interested in that. ’cause he, he was here at the point that we were, we were discussing that. And, um, uh, also, um, a colleague, um, od Petter in, in, um, Oslo and University of Oslo, who I also work with.
So we’re, we’re very keen to kind of take that idea forward.
Duncan: Okay, good. And then so, um, one final question. This is actually from me. So, um, there’s, there’s a lot of, a lot of heat and, and [00:43:00] noise made about chat GPT and code generators and, you know, GitHub copilot and all this kind of stuff. And I sort of thought this paper’s interesting, isn’t it?
Because actually, uh, well, so what, why, how, how, what, what? How to phrase my question. Um, um, how, how do, how is code comprehension affected by all these new tools that we’re seeing that enable students to generate code, you know, say write an algorithm for me that does
Quintin: this Well, they can certainly, they can certainly do that pretty well, can’t they?
Yes. Um, that’s the easy bit. Yeah, so you might, you might argue that we’ve been forced to change our attitude from the code writing to the code understanding. Um, of course, I, I’ve got a horrible feeling that chat, GPT is probably quite good at explaining code as well as it not, um, if you were to ask it to explain a piece of code, I’m, I’m not absolutely sure about that.
Duncan: That’s interesting. I’ve not tried. I’ve not tried that. Um, um, yeah, I, and what level would [00:44:00] it explain it on, you know, would it. You know? Yeah. Okay. Yeah. I, I’ve got, I’ve got in a, in a way, it doesn’t matter who wrote the code, whether it’s a human or a machine, you still have to understand it. Right. Is that, is that fair?
Would you agree with that? That’s, that’s what I feel. That’s one of the things I take away from this paper is that. Actually, you know, there’s a lot of people panicking and saying, oh, well software engineers all gonna be out of jobs because, you know, chat GPT can just write their code for them. But actually you’re never gonna escape this fact that somebody somewhere has to understand what code does.
Quintin: Yeah. I mean, I was, you know, I mean, of course 1, 1 1 has the famous examples of, of. Um, spacecraft back in the sixties that crashed in, you know, the, the, that basically failed or stayed orbiting a planet or whatever because of a semicolon that was left in there. Shouldn’t have been, no. Whatever. I, I was just reading a, a book about the early days of SpaceX and the, the third rocket they sent up was kind of absolutely perfect, apart from one number.
[00:45:00] Which, um, said how long there should be between, uh, stage separation and the next rocket firing or something like that. And, and for, for various reasons. Um. Uh, this number was just ti a tiny bit too small. And so
Duncan: it wasn’t, it wasn’t the classic units problem. Was it, it wasn’t the given in kilometers instead of miles or It was No,
Quintin: they, they, they, they, they hadn’t realized an aspect of their first stage engine, which when, when they shut it off, there was still fuel, an oxidizer as it were in the pipes, and so it just produced a tiny amount of extra thrust.
Um, so that, um, the, the first stage accidentally crashed into the second stage because the second stage hadn’t got away quick enough as you, as you might, you know, and, and, and, and that tiny amount of extra thrust had never been perceived on earth when they’d done test firings because it just, it’s all I’m saying is, you know, Jack, GVT could write you your code, but you know.[00:46:00]
You’re not really just gonna trust chat, g chat, GBT, are you in real situations, uh, you’re gonna pour over that code and make sure it works as you expect. Yeah.
Duncan: Good. Okay. So I think, uh, that’s probably a good note to end on. So thank you Quentin, for joining us today, and thank you for coming to General Club.
I’ll post, uh, in the show notes. I’ll make sure there’s a link to the paper. So, um, I’ll end that. Thank you. Thank you. Thank you very much.
Thank you for listening to The Rest is teaching a podcast for computer science educators and practitioners. You can subscribe or listen to this wherever you get your podcasts. We’d like to thank and acknowledge the Council for Professors and Heads of Computing. That’s cphc.ac.uk who have funded this podcast.
[00:47:00] Production was by podcast.co. Thanks again for listening.