Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

Towards better caption supervision for object detection

Chen, Changjian, Wu, Jing ORCID:, Wang, Xiaohan, Xiang, Shouxing, Zhang, Song-Hai, Tang, Qifeng and Liu, Shixia 2022. Towards better caption supervision for object detection. IEEE Transactions on Visualization and Computer Graphics 28 (4) , pp. 1941-1954. 10.1109/TVCG.2021.3138933

[thumbnail of MutualDetector-postprint-combined.pdf]
PDF - Accepted Post-Print Version
Download (5MB) | Preview


As training high-performance object detectors requires expensive bounding box annotations, recent methods resort to freeavailable image captions. However, detectors trained on caption supervision perform poorly because captions are usually noisy and cannot provide precise location information. To tackle this issue, we present a visual analysis method, which tightly integrates caption supervision with object detection to mutually enhance each other. In particular, object labels are first extracted from captions, which are utilized to train the detectors. Then, the label information from images is fed into caption supervision for further improvement. To effectively loop users into the object detection process, a node-link-based set visualization supported by a multi-type relational co-clustering algorithm is developed to explain the relationships between the extracted labels and the images with detected objects. The co-clustering algorithm clusters labels and images simultaneously by utilizing both their representations and their relationships. Quantitative evaluations and a case study are conducted to demonstrate the efficiency and effectiveness of the developed method in improving the performance of object detectors.

Item Type: Article
Date Type: Publication
Status: Published
Schools: Computer Science & Informatics
Publisher: Institute of Electrical and Electronics Engineers
ISSN: 1077-2626
Date of First Compliant Deposit: 10 January 2022
Date of Acceptance: 21 December 2021
Last Modified: 07 Nov 2023 02:49

Actions (repository staff only)

Edit Item Edit Item


Downloads per month over past year

View more statistics