Real-time 3D-aware portrait video relighting

Cai, Ziqi, Jiang, Kaiwen, Chen, Shu-Yu, Lai, Yukun ORCID: https://orcid.org/0000-0002-2094-5680, Fu, Hongbo, Shi, Boxin and Gao, Lin 2024. Real-time 3D-aware portrait video relighting. Presented at: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 17-21 June 2024. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6221-6231. 10.1109/CVPR52733.2024.00595

Preview

PDF - Accepted Post-Print Version Download (1MB) | Preview

Abstract

Synthesizing realistic videos of talking faces under custom lighting conditions and viewing angles benefits various downstream applications like video conferencing. However, most existing relighting methods are either time-consuming or unable to adjust the viewpoints. In this paper, we present the first real-time 3D-aware method for relighting in-the-wild videos of talking faces based on Neural Radiance Fields (NeRF). Given an input portrait video, our method can synthesize talking faces under both novel views and novel lighting conditions with a photo-realistic and disentangled 3D representation. Specifically, we infer an albedo tri-plane, as well as a shading tri-plane based on a desired lighting condition for each video frame with fast dual-encoders. We also leverage a temporal consistency network to ensure smooth transitions and reduce flickering artifacts. Our method runs at 32.98 fps on consumer-level hardware and achieves state-of-the-art results in terms of reconstruction quality, lighting error, lighting instability, temporal consistency and inference speed. We demonstrate the effectiveness and interactivity of our method on various portrait videos with diverse lighting and viewing conditions.

Item Type:	Conference or Workshop Item (Paper)
Date Type:	Published Online
Status:	Published
Schools:	Computer Science & Informatics
ISBN:	979-8-3503-5301-3
ISSN:	1063-6919
Date of First Compliant Deposit:	2 April 2024
Date of Acceptance:	26 February 2024
Last Modified:	27 Nov 2024 14:29
URI:	https://orca.cardiff.ac.uk/id/eprint/167661

Actions (repository staff only)

Edit Item