NormXLogit: The head-on-top never lies

Abbasi, Sina, Modarres, Mohammad Reza and Pilehvar, Mohammad Taher 2025. NormXLogit: The head-on-top never lies. Presented at: 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, 4-9 November 2025. Published in: Christodoulopoulos, Christos, Chakraborty, Tanmoy, Rose, Carolyn and Peng, Violet eds. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 34914-34935. 10.18653/v1/2025.emnlp-main.1769

PDF - Published Version Available under License Creative Commons Attribution. Download (1MB)

Abstract

With new large language models (LLMs) emerging frequently, it is important to consider the potential value of model-agnostic approaches that can provide interpretability across a variety of architectures. While recent advances in LLM interpretability show promise, many rely on complex, model-specific methods with high computational costs. To address these limitations, we propose NormXLogit, a novel technique for assessing the significance of individual input tokens. This method operates based on the input and output representations associated with each token. First, we demonstrate that the norm of word embeddings can be utilized as a measure of token importance. Second, we reveal a significant relationship between a token’s importance and how predictive its representation is of the model’s final output. Extensive analyses indicate that our approach outperforms existing gradient-based methods in terms of faithfulness and offers competitive performance compared to leading architecture-specific techniques.

Item Type:	Conference or Workshop Item - published (Paper)
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	Association for Computational Linguistics
ISBN:	979-8-89176-332-6
Date of First Compliant Deposit:	16 June 2026
Last Modified:	02 Aug 2026 01:19
URI:	https://orca.cardiff.ac.uk/id/eprint/187582

Actions (repository staff only)

Edit Item