Interactive NeRF geometry editing with shape priors

Yuan, Yu-Jie, Sun, Yang-Tian, Lai, Yu-Kun ORCID: https://orcid.org/0000-0002-2094-5680, Ma, Yuewen, Jia, Rongfei, Kobbelt, Leif and Gao, Lin 2023. Interactive NeRF geometry editing with shape priors. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (12) , pp. 14821-14837. 10.1109/TPAMI.2023.3315068

Preview

PDF - Accepted Post-Print Version Download (27MB) | Preview

Abstract

Neural Radiance Fields (NeRFs) have shown great potential for tasks like novel view synthesis of static 3D scenes. Since NeRFs are trained on a large number of input images, it is not trivial to change their content afterwards. Previous methods to modify NeRFs provide some control but they do not support direct shape deformation which is common for geometry representations like triangle meshes. In this paper, we present a NeRF geometry editing method that first extracts a triangle mesh representation of the geometry inside a NeRF. This mesh can be modified by any 3D modeling tool (we use ARAP mesh deformation). The mesh deformation is then extended into a volume deformation around the shape which establishes a mapping between ray queries to the deformed NeRF and the corresponding queries to the original NeRF. The basic shape editing mechanism is extended towards more powerful and more meaningful editing handles by generating box abstractions of the NeRF shapes which provide an intuitive interface to the user. By additionally assigning semantic labels, we can even identify and combine parts from different objects. We demonstrate the performance and quality of our method in a number of experiments on synthetic data as well as real captured scenes.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	Institute of Electrical and Electronics Engineers
ISSN:	0162-8828
Funders:	The Royal Society
Date of First Compliant Deposit:	20 September 2023
Date of Acceptance:	28 August 2023
Last Modified:	13 Dec 2023 17:59
URI:	https://orca.cardiff.ac.uk/id/eprint/162655

Actions (repository staff only)

Edit Item