APSE: Attention-aware polarity-sensitive embedding for emotion-based image retrieval

Yao, Xingxu, Zhao, Sicheng, Lai, Yu-Kun ORCID: https://orcid.org/0000-0002-2094-5680, She, Dongyu, Liang, Jie and Yang, Jufeng 2021. APSE: Attention-aware polarity-sensitive embedding for emotion-based image retrieval. IEEE Transactions on Multimedia 23 , pp. 4469-4482. 10.1109/TMM.2020.3042664

Preview

PDF - Accepted Post-Print Version Download (1MB) | Preview

Abstract

With the popularity of social media, an increasing number of people are accustomed to expressing their feelings and emotions online using images and videos. An emotion-based image retrieval (EBIR) system is useful for obtaining visual contents with desired emotions from a massive repository. Existing EBIR methods mainly focus on modeling the global characteristics of visual content without considering the crucial role of informative regions of interest in conveying emotions. Further, they ignore the hierarchical relationships between coarse polarities and fine categories of emotions. In this paper, we design an attention-aware polarity-sensitive embedding (APSE) network to address these issues. First, we develop a hierarchical attention mechanism to automatically discover and model the informative regions of interest. Specifically, both polarity-and emotion-specific attended representations are aggregated for discriminative feature embedding. Second, we propose a generated emotion-pair (GEP) loss to simultaneously consider the inter-and intra-polarity relationships of the emotion labels. Moreover, we adaptively generate negative examples of different hard levels in the feature space guided by the attention module to further improve the performance of feature embedding. Extensive experiments on four popular benchmark datasets demonstrate that the proposed APSE method outperforms the state-of-the-art EBIR approaches by a large margin.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	Institute of Electrical and Electronics Engineers (IEEE)
ISSN:	1520-9210
Date of First Compliant Deposit:	19 December 2020
Date of Acceptance:	22 November 2020
Last Modified:	06 Nov 2023 18:17
URI:	https://orca.cardiff.ac.uk/id/eprint/137133

Citation Data

Cited 5 times in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item