CVBench: benchmarking and comparing video generation with large multimodal models

Wang, Jiarui, Duan, Huiyu, Xing, Yuke, Zhou, Wei, Zhai, Guangtao and Min, Xiongkuo 2025. CVBench: benchmarking and comparing video generation with large multimodal models. Presented at: 2025 International Conference on Visual Communications and Image Processing (VCIP), Klagenfurt, Austria, 1-4 December 2025. 2025 International Conference on Visual Communications and Image Processing (VCIP). 2025 International Conference on Visual Communications and Image Processing (VCIP). IEEE, 10.1109/vcip67698.2025.11396889

Full text not available from this repository.

Abstract

Large multimodal models (LMMs) have revolutionized both text-to-video (T2V) generation and video-to-text (V2T) interpretation. However, despite these advancements, issues such as imperfect perceptual quality and inconsistent text-video alignment continue to limit the practical deployment of AI-generated videos (AIGVs). Consequently, there is a pressing need for a reliable benchmark and automatic evaluation framework tailored for AIGVs. To this end, we propose CVBench, the largest and most comprehensive dataset for Comparative Video Benchmarking, including 60K video pairs generated by 30 state-of-the-art T2V models and 600K pairwise comparisons annotated with over 1.7 million human judgments from perspectives of both perceptual quality and text-video correspondence. This dataset enables bidirectional benchmarking and evaluation of both T2V generation models and V2T interpretation models. Based on CVBench, we propose VComp, a novel LMM-based evaluation metric that captures fine-grained quality differences from multiple perspectives for pairwise comparison at both the instance level and model level. Extensive experiments show that VComp achieves state-of-the-art alignment with human preferences. Both the CVBench dataset and VComp metric will be available at https://github.com/IntMeGroup/CVBench.

Item Type:	Conference or Workshop Item - published (Paper)
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	IEEE
ISBN:	979-8-3315-6867-2
ISSN:	2642-9357
Last Modified:	13 Mar 2026 11:15
URI:	https://orca.cardiff.ac.uk/id/eprint/185727

Actions (repository staff only)

Edit Item