Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

Study of spatio-temporal modeling in video quality assessment

Fang, Yuming, Li, Zhaoqian, Yan, Jiebin, Sui, Xiangjie and Liu, Hantao ORCID: https://orcid.org/0000-0003-4544-3481 2023. Study of spatio-temporal modeling in video quality assessment. IEEE Transactions on Image Processing 32 , pp. 2693-2702. 10.1109/TIP.2023.3272480

[thumbnail of Study_of_Spatio-Temporal_print.pdf]
Preview
PDF - Accepted Post-Print Version
Download (22MB) | Preview

Abstract

Video quality assessment (VQA) has received remarkable attention recently. Most of the popular VQA models employ recurrent neural networks (RNNs) to capture the temporal quality variation of videos. However, each long-term video sequence is commonly labeled with a single quality score, with which RNNs might not be able to learn long-term quality variation well. A natural question then arises: What’s the real role of RNNs in learning the visual quality of videos? Does it learn spatio-temporal representation as expected or just aggregating spatial features redundantly? In this study, we conduct a comprehensive study by training a family of VQA models with carefully designed frame sampling strategies and spatio-temporal fusion methods. Our extensive experiments on four publicly available in-the-wild video quality datasets lead to two main findings. First, the plausible spatio-temporal modeling module ( i.e ., RNNs) does not facilitate quality-aware spatio-temporal feature learning. Second, sparsely sampled video frames are capable of obtaining the competitive performance against using all video frames as the input. In other words, spatial features play a vital role in capturing video quality variation for VQA. To our best knowledge, this is the first work to explore the issue of spatio-temporal modeling in VQA.

Item Type: Article
Date Type: Publication
Status: Published
Schools: Computer Science & Informatics
Publisher: Institute of Electrical and Electronics Engineers
ISSN: 1057-7149
Date of First Compliant Deposit: 6 May 2023
Date of Acceptance: 14 April 2023
Last Modified: 19 Nov 2023 10:27
URI: https://orca.cardiff.ac.uk/id/eprint/159299

Actions (repository staff only)

Edit Item Edit Item

Downloads

Downloads per month over past year

View more statistics