Social good or scientific curiosity? Uncovering the research framing behind NLP artefacts

Chamoun, Eric, Ousidhoum, Nedjma, Schlichtkrull, Michael Sejr and Vlachos, Andreas 2025. Social good or scientific curiosity? Uncovering the research framing behind NLP artefacts. Presented at: 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, 4-9 November 2025. Published in: Christodoulopoulos, Christos, Chakraborty, Tanmoy, Carolyn, Rose and Peng, Violet eds. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 25310-25346. 10.18653/v1/2025.emnlp-main.1286

PDF - Published Version Available under License Creative Commons Attribution. Download (2MB)

Abstract

Clarifying the research framing of NLP artefacts (e.g., models, datasets, etc.) is crucial to aligning research with practical applications when researchers claim that their findings have real-world impact. Recent studies manually analyzed NLP research across domains, showing that few papers explicitly identify key stakeholders, intended uses, or appropriate contexts. In this work, we propose to automate this analysis, developing a three-component system that infers research framings by first extracting key elements (means, ends, stakeholders), then linking them through interpretable rules and contextual reasoning. We evaluate our approach on two domains: automated fact-checking using an existing dataset, and hate speech detection for which we annotate a new dataset—achieving consistent improvements over strong LLM baselines. Finally, we apply our system to recent automated fact-checking papers and uncover three notable trends: a rise in underspecified research goals, increased emphasis on scientific exploration over application, and a shift toward supporting human fact-checkers rather than pursuing full automation.

Item Type:	Conference or Workshop Item - published (Paper)
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	Association for Computational Linguistics
ISBN:	979-8-89176-332-6
Date of First Compliant Deposit:	16 June 2026
Last Modified:	16 Jun 2026 09:45
URI:	https://orca.cardiff.ac.uk/id/eprint/187575

Actions (repository staff only)

Edit Item