From granular grief to binary belief: a collaborative optimization of annotation techniques for anti-autistic language

Rizvi, Naba, Morales Flores, Alexis, Rizvi, Mohammad, Ousidhoum, Nedjma and Munyaka, Imani 2025. From granular grief to binary belief: a collaborative optimization of annotation techniques for anti-autistic language. Proceedings of the ACM on Human-Computer Interaction 9 (7) , pp. 1-23. 10.1145/3757478

PDF - Published Version Available under License Creative Commons Attribution. Download (1MB)

Abstract

Annotating text for subjective tasks, such as labeling ableist and anti-autistic texts, is a challenge that has attracted significant attention as commonly adopted annotation paradigms, e.g., using majority voting, fall short in capturing the nuances of hate speech or bias annotations. Labeling ableist and anti-autistic texts presents similar challenges in addition to the need for familiarity with autism and anti-autistic discrimination. In this paper, we adopt a collaborative and annotator-centric approach to study the impact of various annotation techniques. We recruit 6 participants to annotate sets of sentences from our 11,596 sentence corpus. The groups annotate through schemes focused on score-based classification, algorithmic labeling, and comparison-based labeling to identify instances of anti-autistic ableist speech. As a result of changes in annotation schemes, our annotator groups shift from a worse-than-chance agreement to moderate agreement. This suggests that implementing annotator group discussion and collecting annotator feedback is likely to result in improved agreement scores in difficult and highly subjective tasks. Our results highlight the importance of a collaborative approach in highly subjective classification tasks as it may lead to an improved understanding of their own biases, and large improvements in agreement scores, particularly among annotators with higher rates of disagreement. Warning: This paper contains examples that may be offensive or upsetting, including explicit slurs used against people with disabilities.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	Association for Computing Machinery (ACM)
Date of First Compliant Deposit:	29 October 2025
Date of Acceptance:	31 March 2025
Last Modified:	29 Oct 2025 10:15
URI:	https://orca.cardiff.ac.uk/id/eprint/181970

Actions (repository staff only)

Edit Item