Is There a Free Lunch in Inference?

Rouder, Jeffrey N., Morey, Richard D. ORCID: https://orcid.org/0000-0001-9220-3179, Verhagen, Josine, Province, Jordan M. and Wagenmakers, Eric-Jan 2016. Is There a Free Lunch in Inference? Topics in Cognitive Science 8 (3) , pp. 520-547. 10.1111/tops.12214

Preview

PDF - Accepted Post-Print Version Download (311kB) | Preview

Abstract

The field of psychology, including cognitive science, is vexed by a crisis of confidence. Although the causes and solutions are varied, we focus here on a common logical problem in inference. The default mode of inference is significance testing, which has a free lunch property where researchers need not make detailed assumptions about the alternative to test the null hypothesis. We present the argument that there is no free lunch; that is, valid testing requires that researchers test the null against a well-specified alternative. We show how this requirement follows from the basic tenets of conventional and Bayesian probability. Moreover, we show in both the conventional and Bayesian framework that not specifying the alternative may lead to rejections of the null hypothesis with scant evidence. We review both frequentist and Bayesian approaches to specifying alternatives, and we show how such specifications improve inference. The field of cognitive science will benefit because consideration of reasonable alternatives will undoubtedly sharpen the intellectual underpinnings of research.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Psychology
Subjects:	B Philosophy. Psychology. Religion > BF Psychology
Uncontrolled Keywords:	Inference; Philosophy of science; Statistics; Replication crisis
Publisher:	Wiley
ISSN:	1756-8757
Date of First Compliant Deposit:	21 July 2017
Date of Acceptance:	18 May 2016
Last Modified:	17 Nov 2024 04:15
URI:	https://orca.cardiff.ac.uk/id/eprint/102829

Citation Data

Cited 57 times in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item