In Hamburg there are several suspected cases in which Abitur exams are said to have been written using the ChatGPT AI language model. So far, however, this cannot be proven with digital tools.
(Photo: dpa)
Dusseldorf How to recognize texts from ChatGPT? Many teachers are currently asking themselves this question. In Hamburg, for example, several high school graduates are suspected of having used artificial intelligence (AI) in their final exams.
Programs like ChatGPT can provide answers to complex questions within seconds. This makes them a gateway for attempts to deceive in Abitur exams or in company recruitment tests.
According to the responsible school authorities, “irregularities were noticed” when correcting the work. The teachers checked the texts of the examinees using software “that reveals texts generated by ChatGPT”. However, it should not be that easy to convict the cheaters.
The problem: There are tools that can be used to test texts for AI-generated passages. Many of these programs are free up to a limited number of characters. But you shouldn’t let a student fail the Abitur on this basis.
ChatGPT: OpenAI often does not recognize texts from the AI itself
Even OpenAI, the company behind ChatGPT, is very cautious. The current leading company in the development of AI language models presented its “AI Classifier” at the beginning of the year, which is intended to recognize AI-generated texts. However, the developers warned in a statement: “Our classifier is not completely reliable.” That’s putting it positively.
In fact, the AI examiner is quite often wrong. During internal tests, OpenAI submitted both AI-generated and human-written texts to the AI Classifier. The result: The AI Classifier correctly identified only 26 percent of the AI-generated texts as “probably written by the AI”. In 74 percent of the cases, the program did not recognize that a text was generated automatically.
ChatGPT: English AI texts are recognized better
The AI Classifier recognized texts written by humans more reliably. However, according to OpenAI, the program incorrectly classified these texts as AI-generated in nine percent of the cases. In the case of the high school graduates, this means that even if no students cheated with ChatGPT, the program would accuse one in ten of attempting to cheat.
We recommend using the classifier for English text only. OpenAI to release its AI Classifier
The information on the reliability of the AI classifier also only refers to English texts. With other languages, the program is said to be even more wrong. “We recommend using the classifier only for English text,” OpenAI wrote when it was released in early 2023. “It performs significantly worse in other languages.”
AI-Detector and Co.: Which tool recognizes AI texts best?
However, it is worth trying other tools. Popular is GPTZero, which the computer science student Edward Tian developed at Princeton University in the US state of New Jersey. The most reliable free tool is currently the AI Detector from Sapling, on which ex-researchers from Google and the Universities of Berkely and Stanford are said to have worked.
In a test by the editing service Scribbr, the AI-Detector achieved the highest score among the most-used free tools. In order to compare the programs, the testers each presented them with 30 texts that were completely, partially or not at all created by AI or edited with rephrasing software.
>> Read also: Six tips for dealing with ChatGPT in everyday work
The US-Israeli start-up Copyleaks did almost as well as the AI Detector in the Scribbr test. However, this failed when it was supposed to evaluate the Handelsblatt text you are reading. This text was written by our author and not by an AI, as the tool claims.
ChatGPT: Writing style and vocabulary indicate AI texts
Scribbr test winner Winston AI from Canada is only available for English and French so far, and it costs at least twelve dollars a month to use it.
Incidentally, ChatGPT itself recommends users to pay attention to vocabulary and writing style when debunking AI-generated texts. If, for example, a student deviates from his normal style in class work, this could be an indication that he has used ChatGPT. This should be much easier for teachers to recognize than for the software.
More: What you need to know about OpenAI’s AI
First publication: 05/30/2023, 2:45 p.m.
