Neural radiance fields from sparse RGB-D images for high-quality view synthesis

Yuan, Yu-Jie, Lai, Yu-Kun ORCID: https://orcid.org/0000-0002-2094-5680, Huang, Yi-Hua, Kobbelt, Leif and Gao, Lin 2023. Neural radiance fields from sparse RGB-D images for high-quality view synthesis. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (7) , pp. 8713-8728. 10.1109/TPAMI.2022.3232502

Preview

PDF - Accepted Post-Print Version Download (7MB) | Preview

Abstract

The recently proposed neural radiance fields (NeRF) use a continuous function formulated as a multi-layer perceptron (MLP) to model the appearance and geometry of a 3D scene. This enables realistic synthesis of novel views, even for scenes with view dependent appearance. Many follow-up works have since extended NeRFs in different ways. However, a fundamental restriction of the method remains that it requires a large number of images captured from densely placed viewpoints for high-quality synthesis and the quality of the results quickly degrades when the number of captured views is insufficient. To address this problem, we propose a novel NeRF-based framework capable of high-quality view synthesis using only a sparse set of RGB-D images, which can be easily captured using cameras and LiDAR sensors on current consumer devices. First, a geometric proxy of the scene is reconstructed from the captured RGB-D images. Renderings of the reconstructed scene along with precise camera parameters can then be used to pre-train a network. Finally, the network is fine-tuned with a small number of real captured images. We further introduce a patch discriminator to supervise the network under novel views during fine-tuning, as well as a 3D color prior to improve synthesis quality. We demonstrate that our method can generate arbitrary novel views of a 3D scene from as few as 6 RGB-D images. Extensive experiments show the improvements of our method compared with the existing NeRF-based methods, including approaches that also aim to reduce the number of input images.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	Institute of Electrical and Electronics Engineers
ISSN:	0162-8828
Funders:	The Royal Society
Date of First Compliant Deposit:	14 January 2023
Date of Acceptance:	10 December 2022
Last Modified:	06 Nov 2023 16:44
URI:	https://orca.cardiff.ac.uk/id/eprint/155871

Actions (repository staff only)

Edit Item