Understanding LLM performance degradation in multi-instance processing: The roles of instance count and context length

Chen, Jingxuan, Pilehvar, Mohammad Taher and Camacho Collados, Jose ORCID: https://orcid.org/0000-0003-1618-7239 2026. Understanding LLM performance degradation in multi-instance processing: The roles of instance count and context length. Presented at: 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), San Diego, California, United States, 2-7 July 2026. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics. , vol.1 Association for Computational Linguistics, pp. 31851-31884. 10.18653/v1/2026.acl-long.1470

Full text not available from this repository.

Abstract

Users often rely on Large Language Models (LLMs) for processing multiple documents or performing analysis over a number of instances. For example, analysing the overall sentiment of a number of movie reviews requires an LLM to process the sentiment of each review individually in order to provide a final aggregated answer. While LLM performance on such individual tasks is generally high, there has been little research on how LLMs perform when dealing with multi-instance inputs. In this paper, we perform acomprehensive evaluation of the multi-instance processing (MIP) ability of LLMs for tasks in which they excel individually. The results show that all LLMs follow a pattern of slight performance degradation for small numbers of instances (≈20–100), followed by a performance collapse on larger instance counts. Crucially, our analysis shows that while context length is associated with this degradation, the number of instances has a stronger effect on the final results. This finding suggests that when optimising LLM performance for MIP, attention should be paid to both context length and, in particular, instance count.

Item Type:	Conference or Workshop Item - published (Paper)
Status:	Published
Schools:	Schools > Computational & Mathematical Sciences Schools > Computer Science & Informatics
Publisher:	Association for Computational Linguistics
ISBN:	979-8-89176-390-6
Last Modified:	02 Nov 2026 12:16
URI:	https://orca.cardiff.ac.uk/id/eprint/190584

Actions (repository staff only)

Edit Item