Writing with AI
« previous post | next post »
It's been clear for a while that "large language models" can be prompted to fulfill writing assignments, and that LLM detection doesn't work, and that "watermarking" won't come to the rescue. There's lots of on-going published discussion, and even more discussion in real life.
As documented by the MLA-CCCC Joint Task Force on Writing and AI, the conclusion seems to be a combination of bringing AI explicitly into the class, and designing some assignments where students need to function without it.
In one recent example, Joe Moxley has posted the syllabus for his course "Writing with Artificial Intelligence – Syllabus (ENC 3370)".
As I suggested in "LLMs in education: the historical view" (5/1/2023), education will survive the introduction of AI, just as it survived the introduction of writing, despite Plato's skepticism. But maybe oral rhetoric would be a useful alternative to hand-written essays, as an AI-free (or at least AI-reduced) dimension of evaluation (and target of instruction)?
See the discussion at the end of "The LLM-detection boom" (7/7/2023), copied below [image from Ethan Mollick's "The Homework Apocalypse", 7/1/2023]:
In 1999, Wendy Steiner and I taught an experimental undergraduate course on "Human Nature".
Both of us were skeptical of essay-question exams, mostly due lack of trust in our own subjective evaluations. So we decided to experiment with an in-person oral midterm. But as I recall, there were about 80 students in the class — so how to do it?
We circulated in advance a list of a dozen or so sample questions, explaining that the actual questions would be similar to those.
And we scheduled 16 (?) one-hour sessions, each involving five students and both instructors sitting around a small seminar table. We went around the table, asking a question of each student in turn, and following up with redirections or additional questions as appropriate,. We limited each such interaction to 10 minutes, and before going on to the next student, we asked the group for (optional, brief) comments or additions. We both took notes on the process.
This took the equivalent of two full 8-hour days — spread over a week, as I recall — but it went smoothly, and was successful in the sense that our separately-assigned grades were almost exactly in agreement.
There are obvious problems with grades based on this kind of oral Q&A, just as there are problems with evaluating in-class essays, take-home exams, and term papers. And I've never repeated the experiment. But maybe I should.
Cervantes said,
September 1, 2024 @ 7:56 am
In a somewhat smaller class — up to maybe 15 students — I have often had them do conference style presentations. The format of the class is that each student has a relevant project of their own, so they aren't all regurgitating the same material, but the project has to be informed by the overall course content. Of course this takes multiple class sessions, but it's informative for everyone so it's a good use of time.
Jerry Packard said,
September 1, 2024 @ 8:00 am
In my last few years at UIUC, I taught a GenEd course that used a writing program called CPR (Calibrated Peer Review), which basically had (hundreds of) students rate each other’s essays based on a fixed set of structural criteria that were used to both write and evaluate essays. I found that the program worked surprisingly well, but this was pre-AI, so I imagine it would be easy for students to instruct AI to write essays using CPR structural criteria.
Stephen Goranson said,
September 1, 2024 @ 8:24 am
AI-generated text may affect definitions or usage of forgery and ghostwriting and deus ex machina.
Rodger C said,
September 1, 2024 @ 9:52 am
I'm not sure what deus ex machina has to do with it. Diabolus ex machina, maybe.
Michael W said,
September 1, 2024 @ 10:25 am
I work with kids up to high school level, and I've noticed that English teachers seem to be moving toward the 'in-class' essay, although the student typically also has to plan out and maybe also present the information prior to this. Some of this is sound pedagogical technique, but I think there is a move toward reducing the weighting of the essay itself. This has of course been in the works thanks to online cheating availability, but AI is going to push it even farther.
I have also noticed students tending to use LLM-AI as a learning companion when reviewing fact-based material (e.g. history, science). It still has limits there, especially as it tends to oversimplify subjects and interpretations, but that seems a more sensible use of the technology.
charles antaki said,
September 1, 2024 @ 3:12 pm
As I understand it, oral examination, even for very large classes, is the norm in at least some continental European universities; uncertainly in at least some Italian universities – and they somehow manage to process one student at a time.
/df said,
September 2, 2024 @ 4:52 am
"… students tending to use LLM-AI as a learning companion …"
Wouldn't they be better off just reading the Wikipedia articles on which the LLM was trained? Cut out the middle-LLM?
Brandon Seah said,
September 2, 2024 @ 1:39 pm
Oral exams were part of the requirements for my MSc in Germany over a decade ago, and I think are still standard practice. It was new to me, as I went to college in the US, where all my formal assessments were in written form. I remember thinking at the time that it would be difficult to adjudicate should a student dispute their grade. The practice was to have two examiners (or an examiner and an observer, usually a senior graduate student), one of whom would be taking minutes which would be filed with the grade report, something like what is described in Mark Liberman's post.
Brett said,
September 2, 2024 @ 1:57 pm
For the Wave Motion class I took at MIT (18.308), the professor, David Benney offered everyone in the class the option of either writing a term paper on a topic in nonlinear fluid mechanics, or coming to his office for an oral exam. There was no other graded material in the course. It was not an especially large class (about ten or twelve graduate students), but I was surprised that I was the only one who opted for the oral. In fact, the oral "exam" just consisted of Prof. Benney and I talking for an hour, he having stated at the outset that as I had been by far the most active participant in in-class discussions, he felt that I had already shown myself to be on top of the material.
Richard Hershberger said,
September 3, 2024 @ 4:46 am
I have a sanguine take on LLM and student writing. In the past, a term paper that managed grammatical sentences that at least obliquely addressed the topic was sufficiently impressive, even if only marginally coherent, as to rate a gentleman's D. This was because we were so pleased that the student managed to produce those grammatical sentences. With LLMs, a paper with grammatical sentences that obliquely address the topic in a marginally coherent manner is trivially easy. So do give credit for it, any more than we would give credit for a student with a calculator managing to correctly perform arithmetic. Grade for content. If the LLM hallucinates and the student leaves it in, nail them for that, for being a poor user of the tool. If the student can tease a good essay out of the LLM, good for them! They have demonstrated enough command of the subject matter to use the tool effectively. It probably helps if we avoid the most obvious topics, so that the training data doesn't have a large corpus of good essays.
Mark Liberman on writing with AI — LOGiCFACE said,
September 3, 2024 @ 2:39 pm
[…] For Language Log, Mark Liberman wrote a piece on AI in education and writing: […]