Comparing the validity of numerical judgments elicited by direct rating and point allocation: Insights from objectively verifiable perceptual tasks

Bottomley, Paul Andrew ORCID: https://orcid.org/0000-0001-8131-6768 and Doyle, John R. 2013. Comparing the validity of numerical judgments elicited by direct rating and point allocation: Insights from objectively verifiable perceptual tasks. European Journal of Operational Research 228 (1) , pp. 148-157. 10.1016/j.ejor.2013.01.005

Full text not available from this repository.

Abstract

Two popular methods for assigning numerical values to a set of to-be-judged objects in order to capture their relative standing are Direct Rating (DR) and Point Allocation (PA). People using PA distribute a fixed sum of 100 points among the objects, while people using DR rate each object on a fixed scale, typically 0–10, later rescaled to sum to 100. Prior research shows that these methods exhibit distinct profiles when values are ranked from largest to smallest, with DR being more test–retest reliable. But which method best translates people’s inner judgments into outer numerical values (is more valid)? Instead of examining subjective or abstract stimuli, we use objectively verifiable perceptual tasks, namely judgments of line length presented using bar charts. We show that (i) DR is more inter-rater reliable than PA; (ii) DR is more accurate than PA at the individual level; (iii) but there is no difference in accuracy when individual judgments are combined to form group-level estimates; and (iv) DR judgments were improved by using prior knowledge of method bias, whereas PA judgments were not.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Business (Including Economics)
Subjects:	H Social Sciences > HG Finance
Uncontrolled Keywords:	Decision analysis; Methods of value elicitation; Direct rating; Point allocation; Validity; Objective tasks
Publisher:	Elsevier
ISSN:	0377-2217
Last Modified:	24 Oct 2022 09:48
URI:	https://orca.cardiff.ac.uk/id/eprint/42172

Citation Data

Cited 15 times in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item