Multimodal sentiment analysis with image-text interaction network

Zhu, Tong, Li, Leida, Yang, Jufeng, Zhao, Sicheng, Liu, Hantao ORCID: https://orcid.org/0000-0003-4544-3481 and Qian, Jiansheng 2023. Multimodal sentiment analysis with image-text interaction network. IEEE Transactions on Multimedia 25 , pp. 3375-3385. 10.1109/TMM.2022.3160060

Preview

PDF - Accepted Post-Print Version Download (2MB) | Preview

Abstract

More and more users are getting used to posting images and text on social networks to share their emotions or opinions. Accordingly, multimodal sentiment analysis has become a research topic of increasing interest in recent years. Typically, there exist affective regions that evoke human sentiment in an image, which are usually manifested by corresponding words in peoples comments. Similarly, people also tend to portray the affective regions of an image when composing image descriptions. As a result, the relationship between image affective regions and the associated text is of great significance for multimodal sentiment analysis. However, most of the existing multimodal sentiment analysis approaches simply concatenate features from image and text, which could not fully explore the interaction between them, leading to suboptimal results. Motivated by this observation, we propose a new image-text interaction network (ITIN) to investigate the relationship between affective image regions and text for multimodal sentiment analysis. Specifically, we introduce a cross-modal alignment module to capture region-word correspondence, based on which multimodal features are fused through an adaptive cross-modal gating module. Moreover, considering the complementary role of context information on sentiment analysis, we integrate the individual-modal contextual feature representations for achieving more reliable prediction. Extensive experimental results and comparisons on public datasets demonstrate that the proposed model is superior to the state-of-the-art methods.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	Institute of Electrical and Electronics Engineers
ISSN:	1520-9210
Date of First Compliant Deposit:	15 March 2022
Date of Acceptance:	8 March 2022
Last Modified:	06 Nov 2023 21:38
URI:	https://orca.cardiff.ac.uk/id/eprint/148383

Citation Data

Cited 2 times in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item