Investigating marker accuracy in differentiating between student scripts and those produced using ChatGPT

Hassoulas, Athanasios ORCID: https://orcid.org/0000-0002-1029-1847, Powell, Ned, Roberts, Lindsay ORCID: https://orcid.org/0000-0001-7236-1276 and Umla-Runge, Katja ORCID: https://orcid.org/0000-0002-9615-8907 2023. Investigating marker accuracy in differentiating between student scripts and those produced using ChatGPT. Presented at: AI in Education, Cardiff University, 29 June 2023. Journal of Applied Learning & Teaching. Journal of Applied Learning & Teaching, 10.37074/jalt.2023.6.2.13

PDF - Published Version Available under License Creative Commons Attribution. Download (440kB)

Abstract

The introduction of OpenAI’s ChatGPT has widely been considered a turning point for assessment in higher education. Whilst we find ourselves on the precipice of a profoundly disruptive technology, generative artificial intelligence (AI) is here to stay. At present, institutions around the world are considering how best to respond to such new and emerging tools, ranging from outright bans to re-evaluating assessment strategies. In evaluating the extent of the problem that these tools pose to the marking of assessments, a study was designed to investigate marker accuracy in differentiating between scripts prepared by students and those produced using generative AI. A survey containing undergraduate reflective writing scripts and postgraduate extended essays was administered to markers at a medical school in Wales, UK. The markers were asked to assess the scripts on writing style and content, and to indicate whether they believed the scripts to have been produced by students or ChatGPT. Of the 34 markers recruited, only 23% and 19% were able to correctly identify the ChatGPT undergraduate and postgraduate scripts, respectively. A significant effect of suspected script authorship was found for script content, X²(4, n=34) = 10.41, p<0.05, suggesting that written content holds clues as to how markers assign authorship. We recommend consideration be given to how generative AI can be responsibly integrated into assessment strategies and expanding our definition of what constitutes academic misconduct in light of this new technology.

Item Type:	Conference or Workshop Item (Keynote)
Date Type:	Published Online
Status:	Published
Schools:	Medicine Academic & Student Support Service
Subjects:	L Education > LC Special aspects of education > LC5201 Education extension. Adult education. Continuing education
Publisher:	Journal of Applied Learning & Teaching
ISSN:	2591-801X
Date of First Compliant Deposit:	10 June 2024
Date of Acceptance:	22 July 2023
Last Modified:	10 Jun 2024 13:51
URI:	https://orca.cardiff.ac.uk/id/eprint/168957

Actions (repository staff only)

Edit Item