Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

Nine raters, one index: Carrying LLM disagreement into labour-market estimates

Henseke, Golo, Davies, Rhys ORCID: https://orcid.org/0000-0002-3479-625X, Felstead, Alan ORCID: https://orcid.org/0000-0002-8851-4289, Gallie, Duncan, Green, Francis and Zhou, Ying 2027. Nine raters, one index: Carrying LLM disagreement into labour-market estimates. Technological Forecasting and Social Change 236 , 124933. 10.1016/j.techfore.2026.124933

Full text not available from this repository.

Abstract

When a large language model supplies research annotations, the choice of model becomes an analytic degree of freedom. We ask how much that choice matters by putting 1100 task–occupation cells to nine models from three vendors and weighting their ratings with worker-reported task importance in the British Skills and Employment Surveys. Rankings are substantially more stable than levels: pairwise rank correlations range from 0.74 to 0.92, while the share of British jobs scoring above 0.5 ranges from under 0.1% to 38%. We therefore treat the rater as one dimension of a multiverse, reporting each rater-specific estimate and its range alongside a Rubin-style summary that incorporates their observed dispersion. The consequences depend on the downstream empirical design. A post-2022 pay gradient of –0.020 log points per interquartile range is negative under all nine raters, although it continues a pre-existing trend. In online vacancies, all nine raters initially produce negative post-2022 gradients; after teleworkability is given its own period path, seven produce significant positive coefficients, two produce small negative coefficients, and the pooled estimate is indistinguishable from zero. Neither application identifies an effect of generative AI. Validation against worker-reported AI use, competing exposure measures, assistant traffic and an independent task survey supports incremental predictive content validity and portability, but any bias shared across raters remains unidentified. The paper’s contribution is a procedure for exposing and carrying observed LLM-rater disagreement into empirical estimates.

Item Type: Article
Date Type: Publication
Status: Published
Schools: Schools > Social Sciences (Includes Criminology and Education)
Schools > School of Social and Spatial Sciences
Research Institutes & Centres > Wales Institute of Social & Economic Research, Data & Methods (WISERD)
Publisher: Elsevier
ISSN: 0040-1625
Date of Acceptance: 17 October 2026
Last Modified: 16 Nov 2026 10:00
URI: https://orca.cardiff.ac.uk/id/eprint/190918

Actions (repository staff only)

Edit Item Edit Item