Blind men and the elephant: diverse perspectives on gender stereotypes in benchmark datasets

Zakizadeh, Mahdi and Pilehvar, Mohammad 2025. Blind men and the elephant: diverse perspectives on gender stereotypes in benchmark datasets. Presented at: 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, 4-9 November 2025. Published in: Christodoulopoulos, Christos, Chakraborty, Tanmoy, Rose, Carolyn and Peng, Violet eds. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 22838-22851. 10.18653/v1/2025.emnlp-main.1162

PDF - Published Version Available under License Creative Commons Attribution. Download (320kB)

Abstract

Accurately measuring gender stereotypical bias in language models is a complex task with many hidden aspects. Current benchmarks have underestimated this multifaceted challenge and failed to capture the full extent of the problem. This paper examines the inconsistencies between intrinsic stereotype benchmarks. We propose that currently available benchmarks each capture only partial facets of gender stereotypes, and when considered in isolation, they provide just a fragmented view of the broader landscape of bias in language models. Using StereoSet and CrowS-Pairs as case studies, we investigated how data distribution affects benchmark results. By applying a framework from social psychology to balance the data of these benchmarks across various components of gender stereotypes, we demonstrated that even simple balancing techniques can significantly improve the correlation between different measurement approaches. Our findings underscore the complexity of gender stereotyping in language models and point to new directions for developing more refined techniques to detect and reduce bias.

Item Type:	Conference or Workshop Item - published (Paper)
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	Association for Computational Linguistics
ISBN:	979-8-89176-332-6
Date of First Compliant Deposit:	16 June 2026
Last Modified:	16 Jun 2026 11:02
URI:	https://orca.cardiff.ac.uk/id/eprint/187573

Actions (repository staff only)

Edit Item