When can we reuse a calibration set for multiple conformal predictions?

Balinsky, Alexander ORCID: https://orcid.org/0000-0002-8151-4462 and Balinsky, Alexander David 2025. When can we reuse a calibration set for multiple conformal predictions? Proceedings of Machine Learning Research 266 , pp. 34-42.

PDF - Accepted Post-Print Version Download (730kB)

Abstract

Reliable uncertainty quantification is crucial for the trustworthiness of machine learning applications. Inductive Conformal Prediction (ICP) offers a distribution-free framework for generating prediction sets or intervals with user-specified confidence. However, standard ICP guarantees are marginal and typically require a fresh calibration set for each new prediction to maintain their validity. This paper addresses this practical limitation by demonstrating how e -conformal prediction, in conjunction with Hoeffding’s inequality, can enable the repeated use of a single calibration set with a high probability of preserving the desired coverage. Through a case study on the CIFAR-10 dataset, we train a deep neural network and utilise a calibration set to estimate a Hoeffding correction. This correction allows us to apply a modified Markov’s inequality, leading to the construction of prediction sets with quantifiable confidence. Our results illustrate the feasibility of maintaining provable performance in conformal prediction while enhancing its practicality by reducing the need for repeated calibration. The code for this work is publicly available.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Schools > Mathematics
Additional Information:	Author certifies available file is "The version of the manuscript that has been peer-reviewed and accepted but not yet finalised through publisher copy-editing and layout processes" - AB 07/11/2025
ISSN:	2640-3498
Date of First Compliant Deposit:	5 November 2025
Date of Acceptance:	12 August 2025
Last Modified:	07 Nov 2025 15:15
URI:	https://orca.cardiff.ac.uk/id/eprint/182142

Actions (repository staff only)

Edit Item