Vision-language consistency guided multi-modal prompt learning for blind AI generated image quality assessment

Fu, Jun, Zhou, Wei, Jiang, Qiuping, Liu, Hantao ORCID: https://orcid.org/0000-0003-4544-3481 and Zhai, Guangtao 2024. Vision-language consistency guided multi-modal prompt learning for blind AI generated image quality assessment. IEEE Signal Processing Letters 31 , pp. 1820-1824. 10.1109/LSP.2024.3420083

Preview

PDF - Accepted Post-Print Version Download (569kB) | Preview

Abstract

Recently, textual prompt tuning has shown inspirational performance in adapting Contrastive Language-Image Pre-training (CLIP) models to natural image quality assessment. However, such uni-modal prompt learning method only tunes the language branch of CLIP models. This is not enough for adapting CLIP models to AI generated image quality assessment (AGIQA) since AGIs visually differ from natural images. In addition, the consistency between AGIs and user input text prompts, which correlates with the perceptual quality of AGIs, is not investigated to guide AGIQA. In this letter, we propose visionlanguage consistency guided multi-modal prompt learning for blind AGIQA, dubbed CLIP-AGIQA. Specifically, we introduce learnable textual and visual prompts in language and vision branches of CLIP models, respectively. Moreover, we design a text-to-image alignment quality prediction task, whose learned vision-language consistency knowledge is used to guide the optimization of the above multi-modal prompts. Experimental results on two public AGIQA datasets demonstrate that the proposed method outperforms state-of-the-art quality assessment models.

Item Type:	Article
Date Type:	Published Online
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	Institute of Electrical and Electronics Engineers
ISSN:	1070-9908
Date of First Compliant Deposit:	29 June 2024
Date of Acceptance:	21 June 2024
Last Modified:	09 Nov 2024 03:45
URI:	https://orca.cardiff.ac.uk/id/eprint/170176

Actions (repository staff only)

Edit Item