Episode 5: Steve Draper and Joseph Maguire on the Different Types of Contributions to Knowledge
“Science is a broad church, full of narrow minds, trained to know ever more about even less”. That’s according to Steve Jones, but in Computing Education Research (CER) are we being too narrow-minded about what counts (and what doesn’t count) as a contribution? In episode 5 of our podcast we spoke to Steve Draper and Joseph Maguire at the University of Glasgow to discuss their paper published in TOCE on The different types of contributions to knowledge (in CER): All needed, but not all recognised. From the abstract:
The overall aim of this paper is to stimulate discussion about the activities within CER, and to develop a more thoughtful and explicit perspective on the different types of research activity within CER, and their relationships with each other. While theories may be the most valuable outputs of research to those wishing to apply them, for researchers themselves there are other kinds of contribution important to progress in the field. This is what relates it to the immediate subject of this special journal issue on theory in CER. We adopt as our criterion for value “contribution to knowledge”. This paper’s main contributions are: A set of 12 categories of contribution which together indicate the extent of this terrain of contributions to research. Leading into that is a collection of ideas and misconceptions which are drawn on in defining and motivating “ground rules”, which are hints and guidance on the need for various often neglected categories. These are also helpful in justifying some additional categories which make the set as a whole more useful in combination. These are followed by some suggested uses for the categories, and a discussion assessing how the success of the paper might be judged.
Full transcript and selected references below, cite this blog post using DOI:10.59350/4yp18-76n10
References
- Listen at pod.co/the-rest-is-teaching/steve-draper-and-joseph-maguire
- Steve Draper and Joseph Maguire (2022) The different types of contributions to knowledge (in CER): All needed, but not all recognised ACM Transactions on Computing Education (TOCE) DOI:10.1145/3487053
- This paper was discussed at journal club meeting № 28, see DOI:10.59350/sigcse.1541
Episode transcript, machine generated
⚠️ DISCLAIMER: Please note this transcript has been generated using speech-to-text software. It contains transcription errors and speech disfluency. ⚠️
Duncan: [00:00:00] Hello and welcome to the Rest is Teaching podcast for computing education researchers and practitioners. In this podcast, we’ll meet people who are changing the way that we teach computing from school through to university and beyond. What is computer science anyway? Why should people learn it and how can we improve the way that it’s taught?
My name is Duncan. I’m your host and we’ll be meeting authors of studies and papers that tackle these important issues from our journal club. What is their practice and research? Why is it important, and how can their insights be useful to other people teaching computing in any area of education and at any level?
So thank you Joe and Steve for joining us today. We’re discussing your paper. On the different types of [00:01:00] contributions to knowledge in computing, education, research, uh, all needed but not all recognized. And this was published last year in a CM attractions transactions on.
So can you, in a nutshell. I, I know it’s hard, but in, um, can you explain to people who’ve not read it or heard of the paper? What, what the, what the paper is all about? Well, actually before, no, before we do that, no, let, let’s start with that. So, uh, can you explain what, what the paper is all about?
Joseph : Uh, so, so in terms of what the, the, the paper is about in terms of what’s the, the focus.
So we, we put together this paper in terms of trying to think about. What are all the contributions that can be made in computing education, research, and I suppose what, what motivated it or, or prompted it was a lot of different, um, thoughts around [00:02:00] trying to stimulate a discussion or debate about what are the sort of contributions to computing education research here at the University of Glasgow.
We have lots of people engaged. You, you could say computing education, research, but it takes many different forms. And I think in some senses as, as people enter, uh, the, the area, they may look at separate conferences or journals and, and feel like, well, I’m not sure how I contribute here, or I’m not sure if I have something to contribute, or I may have quite a small.
A contribution, or I might have quite a large one. And there was lots of discussion around this in terms of, well, well what does this look like? Um, some people have debates about, well, it has to be very heavily cemented in theory before you can. Submit a, a, a a paper and different ideas and other people say, well, I would like to outline or structure a, a practice or something I, I want to talk about.
And, and this got to me and Steve having sort of drawn out conversations about the idea of, well, what really are [00:03:00] contributions to knowledge and this domain of, uh, computing education, research as relative sort of outsiders. So we’re not, um, at the center of. The, the community in that sense. Steve is from a school of psychology here, sorry, the School of Psychology here at the University of Glasgow.
Um, the, the School of Computing Science, but, but my background is all in computing education research, so it was, it was sort of having these, uh, debates and discussions around that. And it was, it was, it started off, to be honest. Well, well, right. We’ll, we’ll think about this. And I mean, even if it doesn’t go anywhere, it’ll be quite nutritional or valuable for the two of us, because at least we all have some sense of, of what matters and how to articulate it to other, other people.
And that’s really what it started to, to take the form of, of just trying to think about. Well, what are all the different types of contribution? And not to be too, sort of exclusionary to try and understand really how they fit and why they’re important, at least at this point in time.
Duncan: Yeah. Okay. So I, [00:04:00] one thing I realized there is I, I didn’t ask you to introduce yourselves, so you introduced Steve.
Um, so do, do you want to, just, just for the record, do you wanna say, Steve, do you wanna say a little bit about who you are and where you come from, what your experience is?
Steve: Um. Well, um, I have a long mixed background all there. First degree in physics, but a HD and what was then artificial intelligence. But for the last few decades I’ve been focused on theories of learning and teaching and higher education.
So I definitely have a cross discipline perspective. I think. Most disciplines can’t imagine teaching differently than they do, so they don’t read education, so they don’t find the solutions that published years ago or at least the offered solutions, a few of which turn out to apply in other disciplines.
Duncan: Yeah, I mean that was, that was quite interesting actually, what you said in, in the, when we, when we discussed the, the paper, [00:05:00] you were saying that one problem with the theory side of things is that every discipline has its own theory. They’re not really talking to each other. So, uh, the compute scientists have their theories about how people learn, and then mathematicians have got their theories and they’re all completely different, which is a kind of, kind of a little bit a part of the problem, I suppose.
So, um, oh, Joseph, could you, could you also introduce yourself? So you are, you are coming at it more from a, a computer science side, from the, from the perspective of somebody, a practitioner who teaches computing.
Joseph : Yeah. So I’m, I’m Joseph McGuire. I’m at the University of Glasgow. I’m, uh, a senior lecturer, associate professor.
Um, I’m in that sense, I’m, I’ve always been engaged in. Educational was interested me. That’s how I got to, or sorry, that’s how I got to know Steve. It’s, I’m very fortunate I can have these conversations with him here in my institution. And, um, from that aspect of, although my interests are [00:06:00] around areas such as, uh, cybersecurity and aspects such as that, education and how we think about it is, is very important to me.
And, um, and that’s really my interest or draw here.
Duncan: Right. Okay. So you’ve got, you’ve got these, um. 12 categories of contribution, which we can come on to later. I mean, we probably won’t be able to talk about ’em all. Maybe pick out some of the, some of the important ones. Favorite, sorry, a favorite one. Um, but can you say a little bit about, um, how, how you came to write the paper?
I think you were saying it came from a call for papers. Is that right?
Joseph : Yeah, because we’re not that imaginative. Nothing really prompts us unless, you know, someone else directs us for the work. So, so a call for a call for papers came out, uh, for, for from Toast, um, from, uh, uh, from this call, from papers that came out saying, you know, they, they would like, they were putting together a special issue on a theory.
And, [00:07:00] and I looked at this call for papers and I spoke to Steve and I said, you know, the conversations we’ve been having around, you know. Sort of, uh, what it’s to contribute to CR and what this looks like and what’s the importance of theory. I think this could be really interesting and, and some of the I ideas being put forward, and Steve in particular could be really valuable to.
Steve will have a better line than s me, but it is worthwhile to put, not worthwhile, probably more, more than worthwhile. It, it’s more precise to put things into writing than it is to sit and talk about it. So I love to sit and talk because I like to be imprecise. But if you, if you, I. If you sit and you have to put it down into words to make it more precise, it makes you as sharp as your thought.
You could spend time on it. And so this call for papers came out and, and we put in our, our abstract. And, um, there’s a, there is a really lovely article, um, from the editors where they outline, um, the, the editorial of it, of actually the process. And I think I just glanced at it there. I think they said they received, um, around 80.[00:08:00]
Abstract. So you put in your abstract and they would come back and say, look, we’re interested in an article effectively based off of this abstract, you know, the idea you put in your abstract, you put in like roughly what you’re going to do and what the paper’s going to tackle. And then you get feedback saying, okay, I I, I think we can take this forward.
You should put forward a, a full article. And I think if I remember rightly from the article, they, they didn’t receive to, they didn’t expect to receive such a response of sort of at. Um, abstracts in that sense. And I don’t think we really knew that at the time anyway, when they accepted the abstract as such.
And then, and then we went about, uh, writing the paper and then like all good writers, um, it probably came together later than it did earlier in terms of hurtling towards this deadline, whereas we had. We had all the ideas and all the, the elements, we just had to actually put them down. Um, so from that, from that aspect, that’s really what, what caused that.
And I’m really grateful for the fact the call for participation went out because it, [00:09:00] it prompted us to sit down and do the hard work to actually. Um, document it. I think that’s what was really powerful about, about that, uh, call for participation and the special issue.
Duncan: So you are you, when I you are, there are 12 categories of, of contributions.
Um, I suppose the first, ca first category is one that is, there’s probably, would it be fair to say that there’s no, there’s no lack of representation of theory in the first category? Is a theory, a theory about computing education is something that’s, um. There’s already lots of theories out there. Is that, is that fair to say?
Ooh, there’s no, there’s no shortage. I mean, so what I’m driving at here is, um, which of those categories. So alongside things like theory, you’ve got things like, for example, uh, practice reports. So people saying, I taught this and here’s how it went, and here’s what conclusions I’ve tried to draw from it.
Um, there’s another one, which is a review just reviewing what other people have done. And then another category is theory. So were any of these [00:10:00] categories hard to find examples for in when you were writing the paper?
Joseph : It’s a really good question. I think from a, I think if I, if I, I think what’s interesting is, is the examples that prompted the categories. So as we, as we encountered different examples, you go, I don’t, I don’t really, ’cause you could, to be totally honest, you could probably sit and say, well, it’s necess that because that’s how they’ve been labeled by the community going through.
Sort of different things that people put forward. My personal to sort of jump around and, and bring it back to, to your central question ’cause it’s important. My favorite, uh, of the categories is probably the number three, which is ace, which is argument from common experience. I, I suspect that’s probably.
The biggest contribution of this paper because I, I think it puts down into words, um, really something that everyone understands and appreciates, but didn’t really at, at a minimal level, didn’t have a label to give it [00:11:00] in that sense. But it still has value in this sense of, well, we all, we all have an appreciation for this.
We all know this to be the case, but there’s not necessarily a significant or established clear. Evidence that this is the case. So some people might say, well, you can’t articulate that, or you’re not saying that because it’s not a documented theory or an established theory, or it’s, there’s not a large body of work.
But there’s many things we can debate and argue about, where from our own common experience, we would say, well, that is the case. And we would all recognize that as a community. And we all know that. And so, um, I mean, we can build out a lot more than that sort of very. Tiny sort of element of it, but that’s probably one of my favorite categories.
And the other ones would be the, um, unexpected observations, number eight. So I feel as if those two are, are, are the most interesting or the most valuable that come out of the, the categories. I know people say, you know what, it’s like having children. You’re not meant to have any, you know, no favorite [00:12:00] children.
All categories are equally as important. Ally. I, I like the ace and the,
Duncan: but do you think, though, I’m taking those two categories. So do you think there are things that are, are underrepresented in, in, in computing education, research and there’s not enough people making papers about now If you want to go and publish something about you arguing from common experience or if you want to, um, uh, publish an article that’s about, um, unexpected observations.
Is it, is it hard to get that sort of stuff published at the moment? Is that that sort of seems to be the implication of what you’re saying, or, or maybe I’m reading it wrong?
Steve: No, I
Duncan: think that’s, that’s right.
Steve: In fact, I think my first saw the call for papers asking for papers about theory. I had a contrarian reaction, which I thought there was far too much about that series centrism.
Um, but if I was gonna make that stick, I’d have to say what else? There it is. [00:13:00] And then I have to find examples. Um, so it wasn’t impossible to find examples, but we needed to look for them and sometimes had to go to other areas because some other areas of science are much less ashamed of that. Well, you know, biology has no trouble with declaring a new species, even if they didn’t predict it.
Astronomy is full of, in fact, astronomists. So blatant these days that they even say, of course, we had to say what we expected to find in the telescope. Our experience of the last 10 world-changing missions has been, it’s the things we didn’t have a clue about before, which turned out from cameras have turned out to be the most revolutionary.
Now, computer education isn’t like that, but having said that, I know in my own. Personal teaching practice that the things that have most changed my views on education is when a student said something that I can’t [00:14:00] explain because it doesn’t fit my current theoretical framework. And there it is, one innocent and un deliberate observation by a student who show me if an instant I’ve been mening out a whole idea.
Yeah. The live living experience for me in education at least.
Duncan: So, I mean, and is what you, what you said in, at the start, that was interesting because you, you’re talking about in, in a way, there’s too, perhaps too much theory, or at least, I mean, someone, someone on the, I think it was Amber, uh, who joined us on the discussion was talking about folk theories as in, you know, there are some theories that are good and are useful ’cause they help us predict things and they help us explain things.
But there are some theories. Which are less useful because, um, they don’t really help apart from, in that very small set of circumstances that they’ve been used in. So, um, how do you feel about that in terms of, in terms of theory? Is, is have we got too much theory? Uh, uh, and it, there’s [00:15:00] just different practice.
Steve: I think theory is, I think people expect theory to be the valuable thing. Um. And in one sense it is. If you’re an outsider trying to find out what’s been discovered in the medical field, you probably do want things that are not only supported by data, but they’ve got a good theory. But if you’re actually a researcher in the area, you don’t wanna be stuck behind to the stuff that that’s good for the general newspapers.
You wanna be hearing the things that are upsetting on practice, but you don’t know why yet. So. I think writing this paper has clarified that quite a lot, that if you want a journal to support the research process, then it mustn’t be limiting itself to the stuff that’s been finalized. Yeah.
Duncan: And, and it, it would be nice if, um, I mean, I’m, I’m coming in from my very narrow, narrow viewpoint of a practitioner, but it’d be nice if [00:16:00] you could go to the theory and say, okay, this theory, in, in the, in the, in the discussion, you were talking about theory being the most.
Concentrated form of knowledge. So that’s what you teach. You teach the theory because that’s the most useful thing. But the frustration to people coming into computing education, research from the outside might be, well, I’m just teaching first year programming, for example. Um, how do I do it? You know, show me the theory that tells me how I should do what I should do, and then everything will just fall into place and I’ll know I’ve got a good theory, so I’ll know, I can predict what students are gonna do and that kind of stuff.
So, um,
Joseph : I think, I think theory is incredibly, I, I suppose my own narrative on the importance of theory with computing education, research theory is important and all these elements. I suppose to, to go alongside what Steve was saying there, I feel as if, if I was one of the people at the beginning, I dunno of this movement or this area where I want, say, individuals that are not necessarily [00:17:00] interested in education per se, in computing, and I’m trying to, to mount a, a, a beachhead inside my department or, or gain respect from my colleagues.
I would lean on what people appreciate across that, which is theory, theory, theory. So and so, the thing that’s, I feel as if it’s more a confident aspect of a community is when you can do these as, when you can have arguments from common experience because you’ve now got a community, you’ve now got a group of people where we agree on these things.
I mean, there’s an excellent example in the, I would say this is one of the, the authors of the article and the, and the article it talks about, um. The idea of arguing some common experience is pervasive in many parts to philosophy. Going all the way back to Socrates, like the idea of, well, if everybody in the community or everybody in the audience accepts these principles, we can argue and have a good discussion and go before forward it.
If. A group of people that won’t even engage in your community or even come to in terms of the importa of education and computing education research in part. And you want [00:18:00] to bring them in, I would, you know, you would use theory in terms of trying to give yourself credibility. I think the, I think is some, some sense is a, can be a part where you might be, you might still have that mindset, you know, I dunno if you lived through the war.
You come out the other side and it’s like, you know, you don’t let, you don’t waste sugar and you don’t waste food and you don’t do this, you’re better. You’re still got your tin hat on. You, you, you’re terrified that anything that moves if, if you’re trying to defend your institutions in elements, trying to keep everything as strong balance and it can be very difficult to move beyond that.
Say there are other, the world does not going to end if we, if we talk about this or we think about this. Having said that. It’s not a free for all. You don’t just send into, let’s just have a bunch of people writing thought pieces. Uh, we won’t move forward. So I think we’re, it’s, it’s really challenging.
The one thing, and I, I don’t think we do this in an article. I think the key thing about the article is, is we’re really trying to emphasize all these things are [00:19:00] important. There’s no way, it is not to sort of, you know, I feel as if, and we may come onto this later. Uh, part of the hope is people read the article and it it speaks to you.
So someone reads it and says, oh, they, they obviously value theory and other people read it and say, oh, they’re saying, look, they’re saying it’s not all about theory, it’s about this as well. And you would like to think that in the sense of, we, we want to bring in as many voices as as possible. Um, but that’s not from saying that.
Theory is, is amazing and wonderful in that sense. And I do think, um, there’s another strong aspect of the paper where it talks about theory is the best mileage. If you’re a lecturer, you’re going to try and communicate or, or put things forward. Theory is, is, is incredibly. Um, useful in, in that sense. I think what we’d just like to, to elaborate a bit more is these are all other maybe not appreciated areas and I, I, I’d imagine Steve could elaborate on that a bit, a bit more.
Steve: Well, I’ll say one thing actually that emerges from completing science in this respect. [00:20:00] I used to work in human computer interaction and then when the argument was partly one, that it should be taught in departments, people would put together courses. Based around the published theory and yet the one argument that’s at least in those days is really true of human computer interaction is we know almost nothing.
Predictably, you must test your interface on users, start testing at once and keep on relentlessly, and yet, courses would not, um, give that the emphasis. It seemed more natural to lecture theories. At the danger of insufficient pushing of the one method that was really important to get across. Yeah. Yeah.
Duncan: I mean, I, I suppose one of the, one of the useful things about the paper is it, it helps people from outside understand the discipline a bit better in that, um, if you’re not sure, perhaps what you’re doing, where it fits in.
Then it does help to know, all right, these are the kinds of things and yeah, [00:21:00] okay, maybe some of them are overrepresented and some of them are underrepresented, but then at least you know where you are. So, um, y you, and, and I hope, I would hope that it helps help people articulate that, but um,
Steve: I think that’s right.
So I think the categories are an attempt to give both referees and authors a way of saying why this paper is useful.
Duncan: So, um, in sort of relation to that, um, what, what’s the feedback you’ve had on the paper so far? I mean, obviously if this, it’s been published and people have started citing it, but, um, did you get feedback from the reviewers on, on what, what was the feedback from you had from reviewers in, in order to improve it and, uh, um.
Joseph : It was, it was interesting feedback, I suppose in that sense. It was good feedback in the sense of it seemed to be split and in the sense of, um, the, the significance of it, the value of it. You could tell I felt, you know, maybe. Maybe I’m just too [00:22:00] pessimistic when I, when I look over reviews, that there was, there was people, there was people fighting for it in the sense of they thought it was a, a, a, a really interesting article as other people have felt it, it shouldn’t see the light of day.
Um, and this so ly not be published in a journal like, um. From, from that aspect of it, you could say maybe the, but generally, I, I would say that the thing that was the, the response, if you take the reviews, the thing that was um, really rewarding is that every reviewer engaged with it. It wasn’t, you weren’t getting a review that said like, you know, oh, no, no, no.
It was nonsense and so on and so forth, or, this is excellent. They really did provide, yeah. Really thoughtful responses to it that made you think and, and it resulted in the paper that’s now published. The paper is now published is, is much stronger because of all the efforts of of, of the editors and the reviewers who, who responded to it and, and gave good solid, um, back, uh, and on, [00:23:00] on, on what what they felt was strong, weak, weak about it from that point of view.
So the, the reaction to it, I think was, um. It was quite good that, you know, in general, the thing I think was probably I, I, and Steve is the thing I found more interesting is when you talk to people. So this is a long, relatively long process. Um, it, it, it, it takes, it took a bit of time to put all this together and it, it went on for a while.
What was interesting is, you know, people would ask of what sort of things are you working on, what you doing and, and things like that. And you would talk about this and, and, and those reactions were more interesting, if that makes sense. They were more, so you might talk to more of, I dunno, the old guard of, of a community.
And they might say, I mean, who do you think, yeah, what you’re going to write this article and cover and say, they say, well, well, well, no, I’m, we’re, we’re not doing that because that would dedicate me, if that makes sense. And there’d be similar, people say like, well. Make sure to emphasize where you’re coming from, if that makes sense.
You’re not really coming in to, to judge contributions or judge people or do anything that you’re trying to just [00:24:00] understand it. Um, you had other people reaction, you know, people that like you, the way you’re articulating the voice yourself saying. Uh, I’m a practitioner and I feel as if I’m blocked from, from doing this because, because I can’t relate it to theory and, and so on and so forth, that it was as if we were, although you did, I, I, I don’t know.
But Steve, he might be able to talk about it’s more, but when I talked about it, it’s all I’m saying. You know, I’ve got a muffin in one hand and a cup of coffee. Vanilla saying, well, I’m looking at this. I hadn’t really told them much about the paper. And then all of a sudden. They’d conjured this paper out their head.
So the, the topic was very much alive. The ideas, uh, in this were, were very much alive. Uh, you didn’t really have to elaborate. Everybody had their own sort of tuna song in their head about what you were writing about. And, and, and so, and, and to me that means that area or this topic is of interest to that community.
Duncan: Yeah. Yeah. It was good. I mean, it’s good when it, it does generate discussion because then, uh, you know, you’re onto something. Um, um. I [00:25:00] wondered if, uh, so we had, we had questions. I haven’t, I won’t match questions to names, but we had, um, Amber, Brianna, and, and Ibrahim join us on the the journal club discussion of this paper.
And I was just gonna have a look at some of the questions they asked. One of the questions they asked was. How do we solve this problem? So we have this problem of some contributions not counting, and this paper helps in terms of now those people who can’t get their work published can see where it fits.
But how do we generally solve this problem of, um, these contributions, not counting what’s, what are the next steps?
Joseph : Well, well, I, I’ll, I’ll broadly at layman, I think Steve can maybe elaborate on it more because I, I think he’s always been better articulating this particular argument, this part, and, and if he can’t remember doing so that I can elaborate on it for the point of view of, but I really, I do.
I. One thing I would come at from the point of view, but I think it’s partly the responsibility of the author. So I don’t think we should wait for a [00:26:00] central structure to say, here’s all this. I think it can come from, from the bottom up. And, and what I mean by that is I would, I would try and understand, well, what are, what is your contribution?
What, how do you think you’re actually contributing? And to document that, and one in particular that I think would be useful, and you see some authors doing this, is to. Place what they’ve already done and a sort of lineage of contributions. So the work they’re currently publishing, say it was a conference paper, did that start as, did they do like a, a poster at some point before that or did they attend a workshop or really try to put together almost a blueprint or a, or a timeline of, well, how did we get here?
If that makes sense. And, and put that on their website or talk about it in a YouTube video. Do it, try and communicate. To the community and, and to a wider audience that, you know this, it didn’t just fall out of the sky. I’ve been at this for a while, or this is how we got here. And if you do that with one point, say you do that with a journal article or something substantial, a journal article you’ve produced, or a, a conference paper you’ve produced, [00:27:00] and you, you document this lineage of, of how I got here.
Um, it means when you put forward a smaller contribution or say the first steps of that, you can, you can advocate that as being, well, this is just the, the small steps in the journey. One that jumps to my mind, and I mean, I can say this and I could be totally wrong, is if you take Neil Brown’s work, um, or elements around sort of evidence and, and registered reports and things like that from memory, he had a, a poster in icer, uh, the year I think it was, um.
I think it was 2018, I think it was. But he had a poster there and then subsequently, I think he had a, a conference article and then just recently Toy’s had a journal article. It’s all built out. Um, and, and, and created ideas. It’s not just came out of, of, of nowhere. Whether he’s put together a, a. A webpage or something that documents that journey or whether it’s just in conversations with ’em or other people in my head, um, um, that’s what’s happened.
There’s other examples of that [00:28:00] you could pick out and see. Yeah. Where it’s been a long journey, where they’ve, they’ve thought about and, and I think that’s one thing that people could, could do.
Duncan: And you might, so you might, your contribution might start off in one area and then move through to a different area.
Is, is that what you’re saying? So you might start off with. Perhaps some practice, for example, but actually you end up making a contribution by refuting some theory or, um, arguing from common experience or something.
Joseph : Yeah. So your practice that you’ve done that you may end up doing 7, 8, 9 years in a row, maybe.
Um, and every year you’re are people really documenting that, what they’re changing it year to year, why they’ve changed it, what are they doing now? That body of work may in itself be a contribution. It’s not the fact that you ran it for one semester. You may just say, look, here’s some of the things. I’m quickly sharing it.
And that could be a very small contribution that other people could pick up on in terms of a baker contribution. When people talk about, well, how do I. [00:29:00] Get in or how do I put in a more substantial piece of work? And, uh, you know, to be in a journal is like an archival format, you know, I dunno, it should outlast a nuclear bomb or something to that effect.
Then, then if you think of it like that, then, you know, a one, a one semester class report does that really, should that really go into an archive? If that makes sense. But yeah. Yeah. But nine or 10 years worth of work then, probably so, but I’d imagine after nine or 10 years of delivering something. You probably have tried to connect it to theory.
You probably have tried to do this. You probably have thought about it for a long time because it’s went on for so long. And, and that wisdom and experience would be quite valuable. Um, and, and, and, and then reviewers as well as off of could good record.
Duncan: Oh, that, that’s interesting. I, that, that’s something that didn’t come up in a journal discussion.
That’s, I, that’s quite a really interesting point. I mean, you mentioned Neil’s work about, um. Um, or is it the, the reg registering report. Register reports. So what, what would that fit into? Is that, is that a tool? And when you have a category [00:30:00] of tools, um, it’s not really a tool, is it? It, it’s it’s theory in that you, you have to say what you’re doing before you do it and record the saying of it.
Um, so that people can look back and go, ah, but you said this two years ago and now you’re saying this and we need to be explicit about this, this journey that you’ve been on from point A to point B that, oh, sorry
Joseph : Kerry,
Duncan: I’ve got
Joseph : on now. Go on. Oh, that’s interesting because I suppose you’ve got, you’ve got what he’s advocating for and then you’ve got his act, so he’s advocating his contribution is what he’s advocating for.
So he’s, it’s, it’s the same, to be honest. It’s the same thing of if you had to do, if you had to take the articles or we pontificate and said, well, these are the categories and this is the contribution. If we had to put this, uh, paper into a category, what category would it be? If, if, if that makes sense.
And so that’s roughly, that’s towards the end of the paper that’s discussed, and you could say, well, maybe it comes under an argument from [00:31:00] common experience in terms of these are all these different categories and this is how it works in terms of thinking about what Neil’s work on and what he’s doing.
His, his contribution, you might say, in the, in the, the first, um. Instance in terms of what, what he’s trying to do is he’s trying to put forward, uh, what the community should be doing and, and basing that on various things. So he’s, he’s backing, he’s backing that. Different work. But you see his initial, um, it could be, it could be other people might categorize it different, but it could be a, a practitioner’s report in terms of his idea of.
And well, this is how you actually conduct research. This is why you put it in. These are the flaws with it. This is why people, people are changing things and p hacking and all these different things. You could look at it in terms of what’s being argued, but, but then someone can make an argument from that aspect of it.
But then you can make an argument, um, from a different category. And I suppose another thing, and [00:32:00] again, uh, Steve might have more to say in this or to articulate, is that a useful exercise is to sit and think about, well. The, the, the thing I am contributing, the article I’m putting forward is, well, where, what category would it fit into as I try and communicate what this contribution is?
Because if I can articulate what the contribution is, it’s probably more likely to be accepted wherever it’s, wherever it’s going, or I’ll be able to pick the right venue, um,
Duncan: for it. Yes. Right. The right track of the conference or the right kind of journal or whatever it might be. The right. Whether it’s a conference or that kind of thing, or a workshop or.
Yeah.
Joseph : And, and Steve outlines some of these, I think he’s put together an exercise at the end of the article, uh, in terms of, well, how to use, um, this actual article, um, in terms of, well, how do, how do you use these categories we’ve created? And he puts together, um, uh, a useful, um, approach in terms of thinking about, well, [00:33:00] maybe you could start to think about your own work in this way.
Like, well, what is the contribution or what type is it?
Duncan: Yeah. Yeah. Okay, good. Uh, Steve, is there anything you want to add that in terms of, um, talking about solving the problem, then
Steve: following on from that? I think what we should be doing but aren’t, um, that would be a natural proper progression from the TA seed paper would be doing more to support people who are teaching bits of computer science into going beyond.
Naked unanalyzed, um, experience reports into something the next step up because of course, most people who are teaching are overwhelmed by the job and the moment that marking’s over, they really don’t have an appetite for reanalyzing it. Um, so it would be supporting that progress, I think will be a, a worthy community [00:34:00] boosting activity.
But we haven’t tackled that.
Duncan: It does feel, I mean, we talked about this bit, the journal club, it does feel like our discipline is quite immature in that we’ve only been teaching, uh, computing for. 70 years, uh, you know, 60 is only 50 years old, and yet we’ve been teaching maths and physics, uh, which are in some ways similar.
Um, and they, they’ve done loads of studies and longitudinal studies and randomized controlled trials. And, um, well, there’s still lots, presumably that’s not known about how to teach maths and physics. They seem to be streets ahead of us in terms of, and, um, I, I wondered. You know, be, it’d be interesting to know what, what, what could we learn from other disciplines in terms of how, how we do this.
Because then there’s, there’s two things, isn’t there? There’s people teaching physics will often find themselves teaching computing. Or you know, people even, you know, all, all sorts of people are gonna be wanting to teach people computing. So if [00:35:00] we’ve got theories that we can give to people and say, well here’s how you do it, that’s not just useful to our discipline, it’s useful to lots of other disciplines as well.
But we seem to be quite a long way off that at the moment, it seems to me.
Steve: Well, yes, and I think another drawback is that, um, so if you take physicists teaching computing on the one hand. They’re gonna be good at it. I once had a, a very revealing conversation from someone who was giving me a lift and she had a middle management role and she managed a software team and rather Cheekly asked her whether she, uh, preferred to hire people with computer science degree or physics degrees.
And she pause and had a think about that and she said, well, maybe the physicist, because they’re much more. Pragmatic about what really works and the trouble with computer scientists, they come off actually believing what they were taught about software engineering and checking whether a method really worked.[00:36:00]
Duncan: This a sort of religious cult rather than,
Steve: oh, that’s what she was alluding to. Um, but I think the drawback is that physicists are not very different from computer scientists. And that if they think about how they learned and then they assume any students who matter those. With that same unconscious background and that obviously is not a good recipe.
And I think what’s interesting about maths is they’ve got, had more than 2000 years in taking people with no such prior pre Yeah. And being able to train, at least a few percent of them become affected. Accountants and empires and things like that. And,
Duncan: and, and this perhaps is where some of the theory falls down because it, as you say, it’s like, um, it’s been tested on people who are already quite good at computing.
So a lot, you know, if you’re teaching first year programming, um. They’ve done something like it. They might not have done programming, but they’ve done maths, they’ve done lots of maths. They’ve already got that kind of problem solving abstraction ability. [00:37:00] So you’re not doing this pre and posttest thing where we, we do post-testing a lot in university.
We do exams, we see where they’re coming out, but we really, we really don’t know what level they started at. So we don’t know what add adding, what value we’re adding along the way.
Steve: To put it rudely. We have almost no evidence about the effectiveness of course. Because we only do both tests.
Duncan: Yes.
Steve: And then do our best to shame those who fail as if it wasn’t the teacher’s fault.
Well, yeah.
Duncan: Yeah.
Steve: So what’s stretch now is there is a border it take, but I think we’re not in a great position because up to now it’s not been that broad. Therefore we’ve only had to teach the people who self select for it.
Duncan: Yeah. I mean, do you, do you have anything like that Glasgow? Any sort of moves towards pretesting on any, any particular courses?
Any, any, any courses where you are trying to, you know, perhaps not just in computer science, but I thought that that was something that you, you mentioned in the, in the journal club discussion, which is really interesting, which is, you know, doing this, um, or what difference have we made, [00:38:00] um, uh, on any course.
Is anyone doing it in university where they’re pretesting people at the beginning of the course? And then comparing that with what comes out at the end.
Steve: Systematically, I think it is a sector wide, well, an education wide theoretical problem. On the other hand, that idea comes from, uh, physics education because the work they did on, um, um, the forced concept inventory, which then underpinned what’s now called zus Peer instruction.
Yeah, yeah. Resistance to that test took. At least six, nine years to create. But they managed to preserve it so that it would never be used in assessment for Marx. And that means it can still be used on incoming and outgoing students. So it was so
Duncan: misconceptions that that undergraduate physicists have about forces.
Yeah. A [00:39:00] well, well known and well documented. And you, so
Steve: there’s a Tesla and the point is. They can be expressed in a way that even if you know no physics at all, you can answer the question. So he can use it as a pretest for anyone and then you can apply it. And Azar is very amusing about the impact that had, um, in his own practice.
’cause after all, he teaches at Harvard and he, uh, this, this work on the inventory was published in the, uh, uh, journal of the American Physics Association. Essentially every American physics academic. Would get that journal as part of their subscription. So even those who didn’t think of themselves as educators might run into that article.
And there also no reaction was, of course my students won’t be like that. Harvard students. And then he, he was so sure he tried it on them and was humiliated just like a hundred percent as everyone else. Yeah. Right. Unlike everyone else, he been, get snotty about it. He started getting.
Duncan: Okay, good. [00:40:00] So, um, we’ve, we’ve got, we’ve not got much longer left, but is there anything else that you want to discuss about the paper?
I know there’s one thing, there’s one question on left on my list, which was, um, what’s next? So, um, do you have in your minds, um, something that follows on from this or, um, a practice or, um, papers that you’re gonna write or things that you’re gonna do that follow directly on from this paper? We
Steve: don’t want you stealing them.
Duncan: No. Well, I mean broadly you don’t have to give the away. That was a joke.
Steve: There are two things that are seizing me in this last month or two. One is I’ve just read the Itsy Working Group report on, um, what programs are, and that seems to me a significant different take on what computing science is about.
Defining it as program effectively. That is, it’s about programs. [00:41:00] Now that is a different way of thinking about what’s good in computer science, and that must have, um, implications, which, um, I haven’t had any time at all yet to work on. So that really one difference, but perhaps more direct answer to your question is, I, I’m not sure it’s right.
I should reread the Tasi paper we did, but. I feel we probably underplayed the role of qualitative research as opposed to quantitative. Um, and I think there could be important things to say about qualitative research that weren’t really discussed or specifically allowed for, and not inconsistent with the toasty paper, but I think it might, um, it might be worthwhile exploring that.
Yeah. I say that
Duncan: go. Well,
Steve: I was just add, I think nobody needs the justification of quantitative research if they come from a STEM [00:42:00] background. They take that as obviously worth doing, but that’s not true of qualitative research and yet qualitative research is used and it’s sometimes really important in, uh, CER are.
So we, I don’t think anyone’s articulated why that should be. I think many people would agree it can be important. I don’t think that there’s the same articulation of it and that might be worth, worth working out and writing a paper about.
Joseph : I think just to, to follow onto that, I think a natural, pretty much in the same conversations we’re having is.
It would probably go through to the categories and start to think, oh, the ones that are there right now, it’s not to say that, you know, if it is, well, what does a good contribution actually look like in these categories? And then in terms of thinking about that, when you say, well, what’s a good contribution?
It’s probably coming back to some of these ideas that Steve were just mentioning, that, you know, if you think of it from an evidence point of view in terms of having. Um, the actual to make decisions with or to action or do work or build [00:43:00] upon it. Well, what are the qualities, attributes, elements of that?
And as, as Steve wrote the remarks, probably the paper itself is bent, has a, a mindset slightly bent to, or thinking about quantitative research rather necessarily. Qualitative, not completely, but from the point of view of, because qualitative is quite interesting and that it touches, uh, so many aspects including science research, not just committing education research.
And so the thinking about, well, if you were making a contribution and, and this type and what would you like to see and what would that look like, and what should you really be trying to do? And if you had to reflect on your own work and what you’re doing and, and what are you looking for? Um, what would be valuable?
And, and to be honest, I, I suspect that that would be really valuable in improving people’s. Even practice. And what I mean by that is if you’re a, an individual, a, a, an lecturer, a course coordinator, whatever, and you’re ex executing some intervention or, or practice you want to, to do, and you, and you’re thinking, okay, well I’m, [00:44:00] I’m, I’m gonna step above everybody else, I’m actually going to think about.
How am I going to assess or evaluate that this actually had any effect in the dimensions I care about is, is to know, well, what are those dimensions? What, what should you really be looking at? What are the things that that should be tracked or monitored? And and part of that might have to come through documenting them or, or talking to the wider community because if you’re not necessarily engaged with the community, you may not have a sense of, well, what.
What, what should I really be looking for and what should I be tracking and what is a good contribution? If I had to pass this on the article itself, the, the article in question we were talking about. Talks about the idea of theory being a real value to researchers. And I think if you think about practice, when people come to sharing practice, it’s probably less documented.
Well, what makes for a good practice and our practitioners even hitting that. So do practitioners appreciate that? Well, context is quite important. You might have a [00:45:00] practice that works well in first year or does here, but I’m an individual who has a class at a master’s level that has high diversity or, or so on and so forth.
So how do I take this and adapt it? And so what, what are you documenting? What metrics to use to measure things? Things like that I think are probably as well natural extensions of, of, of this work.
Duncan: Yeah. Yeah. Okay. Good. Okay. Um, I think I should wrap things up there because we’ve probably got, um. More than, um, after.
What do you
Joseph : need? What do you need? We’ve bored you today.
Duncan: No, no, no. It’s interesting. I I kind of, there’s, there’s a lot to discuss in the paper and I mean, like, so we’ve only talked about some of the categories. Um, so I, I’ll encourage people to go and read the paper, right. I really enjoyed reading it. Um, and it’s one of those papers that you go back to and, and, and look at things again sometimes and think, oh, yeah.
The thing that didn’t, the thing you didn’t notice, the first time round sometimes hits you second or third time round. Um, but, um, thank you both [00:46:00] for coming. Um, thank you for writing the paper. Thank you for coming for Jour Journal Club and, um. Um, we’ll see you at the, the next journal, club, whenever that is.
So,
Joseph : definitely, and thank you Duncan for, for offering us the opportunity to come talk about this and it’s, it’s much appreciated.
Duncan: Great. And I’ll post the links to the paper in, in the podcast when that goes live as well. Thank you so much.
Thank you for listening to The Rest is teaching a podcast for computer science educators and practitioners. You can subscribe or listen to this wherever you get your podcasts. We’d like to thank and acknowledge the Council for Professors and Heads of Computing. That’s cpc.ac.uk who have funded this podcast.
Production was by [00:47:00] podcast.co. That’s podcast.co. Thanks again for listening.