NeRF-Texture: Synthesizing Neural Radiance Field textures

Huang, Yi-Hua, Cao, Yan-Pei, Lai, Yu-Kun ORCID: https://orcid.org/0000-0002-2094-5680, Shan, Ying and Gao, Lin 2024. NeRF-Texture: Synthesizing Neural Radiance Field textures. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (9) , pp. 5986-6000. 10.1109/TPAMI.2024.3382198

Preview

PDF - Accepted Post-Print Version Download (16MB) | Preview

Abstract

Texture synthesis is a fundamental problem in computer graphics that would benefit various applications. Existing methods are effective in handling 2D image textures. In contrast, many real-world textures contain meso-structure in the 3D geometry space, such as grass, leaves, and fabrics, which cannot be effectively modeled using only 2D image textures. We propose a novel texture synthesis method with Neural Radiance Fields (NeRF) to capture and synthesize textures from given multi-view images. In the proposed NeRF texture representation, a scene with fine geometric details is disentangled into the meso-structure textures and the underlying base shape. This allows textures with meso-structure to be effectively learned as latent features situated on the base shape, which are fed into a NeRF decoder trained simultaneously to represent the rich view-dependent appearance. Using this implicit representation, we can synthesize NeRF-based textures through patch matching of latent features. However, inconsistencies between the metrics of the reconstructed content space and the latent feature space may compromise the synthesis quality. To enhance matching performance, we further regularize the distribution of latent features by incorporating a clustering constraint. In addition to generating NeRF textures over a planar domain, our method can also synthesize NeRF textures over curved surfaces, which are practically useful. Experimental results and evaluations demonstrate the effectiveness of our approach.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	Institute of Electrical and Electronics Engineers
ISSN:	0162-8828
Date of First Compliant Deposit:	8 May 2024
Date of Acceptance:	10 March 2024
Last Modified:	04 Sep 2024 13:46
URI:	https://orca.cardiff.ac.uk/id/eprint/168793

Actions (repository staff only)

Edit Item