R2Human: Real-time 3D human appearance rendering from a single image

Yang, Yuanwang, Feng, Qiao, Lai, Yukun ORCID: https://orcid.org/0000-0002-2094-5680 and Li, Kun 2024. R2Human: Real-time 3D human appearance rendering from a single image. Presented at: IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Seattle, USA, 21-25 October 2024.

Preview

PDF - Accepted Post-Print Version Download (7MB) | Preview

Abstract

Rendering 3D human appearance from a single image in real-time is crucial for achieving holographic communication and immersive VR/AR. Existing methods either rely on multi-camera setups or are constrained to offline operations. In this paper, we propose R2Human, the first approach for real-time inference and rendering of photorealistic 3D human appearance from a single image. The core of our approach is to combine the strengths of implicit texture fields and explicit neural rendering with our novel representation, namely Z-map. Based on this, we present an end-toend network that performs high-fidelity color reconstruction of visible areas and provides reliable color inference for occluded regions. To further enhance the 3D perception ability of our network, we leverage the Fourier occupancy field as a prior for generating the texture field and providing a sampling surface in the rendering stage. We also propose a consistency loss and a spatial fusion strategy to ensure the multi-view coherence. Experimental results show that our method outperforms the state-of-the-art methods on both synthetic data and challenging real-world images, in real-time. The project page can be found at http://cic.tju. edu.cn/faculty/likun/projects/R2Human.

Item Type:	Conference or Workshop Item (Paper)
Status:	In Press
Schools:	Computer Science & Informatics
Date of First Compliant Deposit:	7 September 2024
Date of Acceptance:	31 July 2024
Last Modified:	03 Oct 2024 10:00
URI:	https://orca.cardiff.ac.uk/id/eprint/171913

Actions (repository staff only)

Edit Item