Integrating large foundation models into multimodal named entity recognition with evidential fusion

Liu, Weide, Zhong, Xiaoyang, Hou, Jingwen, Li, Shaohua, Huang, Haozhe, Zhou, Wei and Fang, Yuming 2025. Integrating large foundation models into multimodal named entity recognition with evidential fusion. Neurocomputing 652 , 131015. 10.1016/j.neucom.2025.131015

Full text not available from this repository.

Abstract

Multimodal Named Entity Recognition (MNER) is a crucial task for information extraction from social media platforms such as Twitter. Most current methods rely on attention weights to extract information from both text and images but are often unreliable and lack interpretability. To address this problem, we propose incorporating uncertainty estimation into the MNER task, producing trustworthy predictions. Our proposed algorithm models the distribution of each modality as a Normal-inverse Gamma distribution, and fuses them into a unified distribution with an evidential fusion mechanism, enabling hierarchical characterization of uncertainties and promotion of prediction accuracy and trustworthiness. Additionally, we explore the potential of pre-trained large foundation models in MNER and propose an efficient fusion approach that leverages their robust feature representations. Experiments on two datasets demonstrate that our proposed method outperforms the baselines and achieves new state-of-the-art performance.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	Elsevier
ISSN:	0925-2312
Date of Acceptance:	13 July 2025
Last Modified:	12 Dec 2025 11:00
URI:	https://orca.cardiff.ac.uk/id/eprint/183161

Actions (repository staff only)

Edit Item