Study of spatio-temporal modeling in video quality assessment

Fang, Yuming, Li, Zhaoqian, Yan, Jiebin, Sui, Xiangjie and Liu, Hantao ORCID: https://orcid.org/0000-0003-4544-3481 2023. Study of spatio-temporal modeling in video quality assessment. IEEE Transactions on Image Processing 32 , pp. 2693-2702. 10.1109/TIP.2023.3272480

Preview

PDF - Accepted Post-Print Version Download (22MB) | Preview

Abstract

Video quality assessment (VQA) has received remarkable attention recently. Most of the popular VQA models employ recurrent neural networks (RNNs) to capture the temporal quality variation of videos. However, each long-term video sequence is commonly labeled with a single quality score, with which RNNs might not be able to learn long-term quality variation well. A natural question then arises: What’s the real role of RNNs in learning the visual quality of videos? Does it learn spatio-temporal representation as expected or just aggregating spatial features redundantly? In this study, we conduct a comprehensive study by training a family of VQA models with carefully designed frame sampling strategies and spatio-temporal fusion methods. Our extensive experiments on four publicly available in-the-wild video quality datasets lead to two main findings. First, the plausible spatio-temporal modeling module ( i.e ., RNNs) does not facilitate quality-aware spatio-temporal feature learning. Second, sparsely sampled video frames are capable of obtaining the competitive performance against using all video frames as the input. In other words, spatial features play a vital role in capturing video quality variation for VQA. To our best knowledge, this is the first work to explore the issue of spatio-temporal modeling in VQA.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	Institute of Electrical and Electronics Engineers
ISSN:	1057-7149
Date of First Compliant Deposit:	6 May 2023
Date of Acceptance:	14 April 2023
Last Modified:	14 Nov 2024 03:45
URI:	https://orca.cardiff.ac.uk/id/eprint/159299

Citation Data

Cited 6 times in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item