Does AI write good science, and does it know whether it can? Evaluating the ability of Generative AI to write and evaluate scientific text

Francis, Nigel ORCID: https://orcid.org/0000-0002-4706-4795 and Shore, Andrew ORCID: https://orcid.org/0000-0001-7115-2050 2025. Does AI write good science, and does it know whether it can? Evaluating the ability of Generative AI to write and evaluate scientific text. Presented at: ASE 2025 Annual Conference, Nottingham, UK, 9 -11 January 2025.

PDF - Presentation Download (1MB)

Abstract

Our project assesses the ability of Generative AI in two key areas. 1. How well does Generative AI construct evidence based scientific text compared to humans? Our assessment includes the impact of different of AI tools, length of prose and few-shot prompting. 2. How well can AI grade and provide feedback against a set of criteria when compared to human assessors. This includes an evaluation of the ability of AI to assess its own product.

Item Type:	Conference or Workshop Item (Paper)
Date Type:	Published Online
Status:	Published
Schools:	Biosciences
Date of Acceptance:	29 July 2024
Last Modified:	14 Jan 2025 17:45
URI:	https://orca.cardiff.ac.uk/id/eprint/175263

Actions (repository staff only)

Edit Item