If you are on Twitter you’ll know that at the moment the only two topics of conversation are “Is Elon Musk a big man-baby?” and “Is ChatGPT evidence that artificial intelligence (more properly ‘machine learning’) is ready to become mainstream?”.
Now, obviously, I have no opinion of the former question but as a teacher I’m clearly interested in the latter. I have long argued that technology can support teachers in their work and looked forward to a time when much of the drudge work can be off-loaded to a digital assistant. ChatGPT seems to make a step forward, not just in bringing that day closer, but also in the range of things it will be able assist with.
First, what is ChatGPT? Let’s ask the beast itself.
Perhaps that’s a little technical for you. Let’s see if it can do better:
Or maybe you’d like an answer that you could use at school with year 6:
So as I understand it, it’s been fed a large cache of text, been taught how to interprete natural language (a la Alexa) and can use that cache of information to feed answers back to you.
First thing to understand, very clearly, is that it’s not intelligent. But it’s able to give the appearance of being intelligent, as long as you stay within the parameters of its programming and its data set. Which makes it not that unlike me! Unfortunately we don’t really know the parameters of its programming, not do we know the initial data of text it’s been fed with. So we have to take care.
So what can it do to help me, a humble Mathematics and Computer Science teacher?
Let’s see if it can do some very simple things for maths.
Immediately we see the limitation. ChatGPT is a chat bot. It’s clearly not a maths bot! It understands what I want, it goes a good way to providing it, but it can’t do the maths (the correct answer is 12 btw).
So I quickly decided to put the maths aside and try the Computer Science. Here we fared a good deal better, being back on the safer ground of textual analysis.
I thought that was a bit technical and didn’t really hit the key points my students need to learn. So I fed ChatGPT with the text from the relevant BBC BiteSize page and asked it to produce a multiple choice quiz, 5 questions each with 4 answers. This was the result:
Ok, now I think we’re getting somewhere. It can, given the right information, provide me with something useful. Which had me thinking about the biggest issue my CompSci student have, which is writing good 8 mark question answers. Students sit 2 papers. On each paper there is one 8 mark essay style question. They struggle with these as the information provided in the question is quite sparse and they have to structure the answer as a form of discussion. So, could ChatGPT answer such a question? Let’s see.
Here’s the first attempt:
Now, the mark scheme for such questions has two dimensions. The first is about the structure:
Now bearing in mind that the student has 10-12 minutes to answer this question, I would put that answer in the high 3-5 mark range or in the low 6-8 mark range. There’s a demonstration of reasonable to thorough knowledge, it’s accurate and there’s a theme running through. The only thing that may militate against 6-8 is a lack of weighing up of both sides, mainly because it hasn’t really covered both sides of the argument..
So I tried it again, this time feeding the mark scheme in as part of the question. This time I got the following answer:
Then (and this is the bit where in my head I went “wow!”) I asked it what mark it would give the first answer it gave:-
The rationale was unprompted, it just thought (sic) that it should give it. For me the scary thing was how closely it matched my own thinking. But went deeper into the specifics. Not only can it mark the questions, it can give good feedback.
So I devised a little test. I got my CompSci class to answer this question. We went over the structure matrix in some detail. We also discussed what kind of issues could be construed as ethical, cultural (and the difficulty of deciding which is which) and environmental issues. I gave them 20 minutes to answer the question (mainly because they were typing rather than writing the answers). Then I marked them (and had a colleague moderate a sample).
Then I fed first the full mark scheme (including the suggested detail points in each area) and then the answers into ChatGPT.
There were 18 answers. For 10 it agreed with the mark band I put them in. The other 8 were all in the mark band below the one I put the answer in. So where there was disagreement it was all consistently in one direction. And, to be honest, some of the feedback given did make me consider if it was me that had got it wrong rather than ChatGPT. The main thing here is the workload it could take off me.
It also provided correct marking of answers across all the mark bands, with relevant feedback.
On the basis of what I’ve found how would I use a resource like ChatGPT?
Based on what I have done (and also what I’ve seen others do) it can certainly be used to provide a range of exemplar answers to questions. For example:
It is also good at creating quick MCQ quizzes, though I need to explore this some more. I have also seen online that it can produce some fairly good Cloze paragraphs). Where it will make an impact is in marking. I am convinced that with a little more training this system could accurately mark these 8 mark questions. As importantly it produced feedback that would be useful to a student, with more detail than I could include without working for an extra several hours. So as a workload assistant ChatGPT does the job.
I have read some criticism of the system which broadly falls into three areas.
Firstly the “it’s not perfect” argument. Well, sorry to break it to you, neither are we. What is also important here is that where it did err, it erred consistently in the same direction.
Secondly we have the “I’m a professor of physics and it got a lot of stuff about my specialism wrong” argument. Well, it’s a test system. Loaded with generic text. Let’s see what happens when its loaded with your specialism.
Thirdly the usual “OMIGOD THE KIDS WILL USE IT FOR THEIR HOMEWORK AND I WON’T KNOW!!!” argument. Well, yes, I’m sure they will. Then you’ll have to do what math’s teachers have been doing for years and start to ask questions that can’t easily be solve by the calculator. But, there is an issue here which will need to be addressed.
In summary, I think this shows how AI can and will change education. If the educators don’t use it then those being educated will. This can be developed to provide resource generation support for teachers. It can also provide some help with marking (in some circumstances, though I’m convinced this will spread to a wider more general use) and it will support the students with greater and (perhaps) more consistent feedback.
The future is nearly here and its come in peace.
Meanwhile, i’ll let ChatGPT have the last word…
It knows what it is, knows its limitations and recognises the humans have to be in control.
For now…
And finally, finally…
You can try ChatGPT for yourself at https://chat.openai.com Requires a simple sign up then you’re away.
Distant Ramblings on the Horizon 