Inferring prototypes for multi-label few-shot image classification with word vector guided attention

Yan, Kun, Zhang, Chenbin, Hou, Jun, Wang, Ping, Bouraoui, Zied, Jameel, Shoaib and Schockaert, Steven ORCID: https://orcid.org/0000-0002-9256-2881 2022. Inferring prototypes for multi-label few-shot image classification with word vector guided attention. Presented at: 36th AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada, 22 February - 01 March 2022. Proceedings of the AAAI Conference on Artificial Intelligence. , vol.36(3) Association for the Advancement of Artificial Intelligence, pp. 2991-2999. 10.1609/aaai.v36i3.20205

Preview

PDF - Accepted Post-Print Version Download (1MB) | Preview

Abstract

Multi-label few-shot image classification (ML-FSIC) is the task of assigning descriptive labels to previously unseen images, based on a small number of training examples. A key feature of the multi-label setting is that images often have multiple labels, which typically refer to different regions of the image. When estimating prototypes, in a metric-based setting, it is thus important to determine which regions are relevant for which labels, but the limited amount of training data makes this highly challenging. As a solution, in this paper we propose to use word embeddings as a form of prior knowledge about the meaning of the labels. In particular, visual prototypes are obtained by aggregating the local feature maps of the support images, using an attention mechanism that relies on the label embeddings. As an important advantage, our model can infer prototypes for unseen labels without the need for fine-tuning any model parameters, which demonstrates its strong generalization abilities. Experiments on COCO and PASCAL VOC furthermore show that our model substantially improves the current state-of-the-art.

Item Type:	Conference or Workshop Item (Paper)
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	Association for the Advancement of Artificial Intelligence
Date of First Compliant Deposit:	4 January 2022
Date of Acceptance:	1 December 2021
Last Modified:	19 May 2025 13:59
URI:	https://orca.cardiff.ac.uk/id/eprint/146320

Actions (repository staff only)

Edit Item