Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

Robust pose transfer with dynamic details using neural video rendering

Sun, Yang-Tian, Huang, Hao-Zhi, Wang, Xuan, Lai, Yukun ORCID: https://orcid.org/0000-0002-2094-5680, Liu, Wei and Gao, Lin 2022. Robust pose transfer with dynamic details using neural video rendering. IEEE Transactions on Pattern Analysis and Machine Intelligence 10.1109/tpami.2022.3166989

[thumbnail of RobustPoseTransfer_TPAMI2022.pdf]
Preview
PDF - Accepted Post-Print Version
Download (23MB) | Preview

Abstract

Pose transfer of human videos aims to generate a high fidelity video of a target person imitating actions of a source person. A few studies have made great progress either through image translation with deep latent features or neural rendering with explicit 3D features. However, both of them rely on large amounts of training data to generate realistic results, and the performance degrades on more accessible internet videos due to insufficient training frames. In this paper, we demonstrate that the dynamic details can be preserved even trained from short monocular videos. Overall, we propose a neural video rendering framework coupled with an image-translation-based dynamic details generation network (D2G-Net), which fully utilizes both the stability of explicit 3D features and the capacity of learning components. To be specific, a novel hybrid texture representation is presented to encode both the static and pose-varying appearance characteristics, which is then mapped to the image space and rendered as a detail-rich frame in the neural rendering stage. Through extensive comparisons, we demonstrate that our neural human video renderer is capable of achieving both clearer dynamic details and more robust performance even on accessible short videos with only 2k-4k frames.

Item Type: Article
Date Type: Published Online
Status: In Press
Schools: Computer Science & Informatics
Publisher: Institute of Electrical and Electronics Engineers
ISSN: 0162-8828
Funders: The Royal Society
Date of First Compliant Deposit: 12 April 2022
Date of Acceptance: 27 March 2022
Last Modified: 10 Nov 2022 17:53
URI: https://orca.cardiff.ac.uk/id/eprint/149168

Actions (repository staff only)

Edit Item Edit Item

Downloads

Downloads per month over past year

View more statistics