Fang, Yuming, Li, Zhaoqian, Yan, Jiebin, Sui, Xiangjie and Liu, Hantao ORCID: https://orcid.org/0000-0003-4544-3481 2023. Study of spatio-temporal modeling in video quality assessment. IEEE Transactions on Image Processing 32 , pp. 2693-2702. 10.1109/TIP.2023.3272480 |
Preview |
PDF
- Accepted Post-Print Version
Download (22MB) | Preview |
Abstract
Video quality assessment (VQA) has received remarkable attention recently. Most of the popular VQA models employ recurrent neural networks (RNNs) to capture the temporal quality variation of videos. However, each long-term video sequence is commonly labeled with a single quality score, with which RNNs might not be able to learn long-term quality variation well. A natural question then arises: What’s the real role of RNNs in learning the visual quality of videos? Does it learn spatio-temporal representation as expected or just aggregating spatial features redundantly? In this study, we conduct a comprehensive study by training a family of VQA models with carefully designed frame sampling strategies and spatio-temporal fusion methods. Our extensive experiments on four publicly available in-the-wild video quality datasets lead to two main findings. First, the plausible spatio-temporal modeling module ( i.e ., RNNs) does not facilitate quality-aware spatio-temporal feature learning. Second, sparsely sampled video frames are capable of obtaining the competitive performance against using all video frames as the input. In other words, spatial features play a vital role in capturing video quality variation for VQA. To our best knowledge, this is the first work to explore the issue of spatio-temporal modeling in VQA.
Item Type: | Article |
---|---|
Date Type: | Publication |
Status: | Published |
Schools: | Computer Science & Informatics |
Publisher: | Institute of Electrical and Electronics Engineers |
ISSN: | 1057-7149 |
Date of First Compliant Deposit: | 6 May 2023 |
Date of Acceptance: | 14 April 2023 |
Last Modified: | 19 Nov 2023 10:27 |
URI: | https://orca.cardiff.ac.uk/id/eprint/159299 |
Citation Data
Cited 6 times in Scopus. View in Scopus. Powered By Scopus® Data
Actions (repository staff only)
Edit Item |