Efficient precision and recall metrics for assessing generative models using hubness-aware sampling

Liang, Yuanbang ORCID: https://orcid.org/0009-0000-8370-6655, Wu, Jing ORCID: https://orcid.org/0000-0001-5123-9861, Lai, Yukun ORCID: https://orcid.org/0000-0002-2094-5680 and Qin, Yipeng ORCID: https://orcid.org/0000-0002-1551-9126 2024. Efficient precision and recall metrics for assessing generative models using hubness-aware sampling. Presented at: The Forty-first International Conference on Machine Learning (ICML), Vienna, Austria, 21-27 July 2024. Proceedings of Machine Learning Research. , vol.235 pp. 29682-29699.

Preview

PDF - Accepted Post-Print Version Download (1MB) | Preview

Abstract

Despite impressive results, deep generative models require massive datasets for training. As dataset size increases, effective evaluation metrics like precision and recall (P&R) become computationally infeasible on commodity hardware. In this paper, we address this challenge by proposing efficient P&R (eP&R) metrics that give almost identical results as the original P&R but with much lower computational costs. Specifically, we identify two redundancies in the original P&R: i) redundancy in ratio computation and ii) redundancy in manifold inside/outside identification. We find both can be effectively removed via hubness-aware sampling, which extracts representative elements from synthetic/real image samples based on their hubness values, i.e., the number of times a sample becomes a k-nearest neighbor to others in the feature space. Thanks to the insensitivity of hubness-aware sampling to exact k-nearest neighbor (k-NN) results, we further improve the efficiency of our eP&R metrics by using approximate k-NN methods. Extensive experiments show that our eP&R matches the original P&R but is far more efficient in time and space. Our code is available at: https://github.com/Byronliang8/Hubness_Precision_Recall

Item Type:	Conference or Workshop Item - published (Paper)
Date Type:	Publication
Status:	Published
Schools:	Professional Services > Advanced Research Computing @ Cardiff (ARCCA) Schools > Computer Science & Informatics
ISSN:	2640-3498
Date of First Compliant Deposit:	24 June 2024
Date of Acceptance:	2 May 2024
Last Modified:	19 Feb 2026 11:35
URI:	https://orca.cardiff.ac.uk/id/eprint/169372

Actions (repository staff only)

Edit Item