Lu, Yuqin, Deng, Bailin ORCID: https://orcid.org/0000-0002-0158-7670, Zhong, Zhixuan, Zhang, Tianle, Quan, Yuhui, Cai, Hongmin and He, Shengfeng 2024. 3D snapshot: Invertible embedding of 3D neural representations in a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence 10.1109/TPAMI.2024.3411051 |
Preview |
PDF
- Accepted Post-Print Version
Download (9MB) | Preview |
Preview |
PDF
- Supplemental Material
Download (14MB) | Preview |
Video (MPEG)
- Supplemental Material
Download (39MB) |
Abstract
3D neural rendering enables photo-realistic reconstruction of a specific scene by encoding discontinuous inputs into a neural representation. Despite the remarkable rendering results, the storage of network parameters is not transmission-friendly and not extendable to metaverse applications. In this paper, we propose an invertible neural rendering approach that enables generating an interactive 3D model from a single image (i.e., 3D Snapshot). Our idea is to distill a pre-trained neural rendering model (e.g., NeRF) into a visualizable image form that can then be easily inverted back to a neural network. To this end, we first present a neural image distillation method to optimize three neural planes for representing the original neural rendering model. However, this representation is noisy and visually meaningless. We thus propose a dynamic invertible neural network to embed this noisy representation into a plausible image representation of the scene. We demonstrate promising reconstruction quality quantitatively and qualitatively, by comparing to the original neural rendering model, as well as video-based invertible methods. On the other hand, our method can store dozens of NeRFs with a compact restoration network (5MB), and embedding each 3D scene takes up only 160KB of storage. More importantly, our approach is the first solution that allows embedding a neural rendering model into image representations, which enables applications like creating an interactive 3D model from a printed image in the metaverse.
Item Type: | Article |
---|---|
Date Type: | Published Online |
Status: | In Press |
Schools: | Computer Science & Informatics |
Subjects: | Q Science > QA Mathematics > QA75 Electronic computers. Computer science Q Science > QA Mathematics > QA76 Computer software |
Publisher: | Institute of Electrical and Electronics Engineers |
ISSN: | 0162-8828 |
Funders: | Guangdong Natural Science Funds for Distinguished Young Scholar, National Research Foundation Singapore |
Date of First Compliant Deposit: | 9 June 2024 |
Date of Acceptance: | 29 May 2024 |
Last Modified: | 08 Oct 2024 01:32 |
URI: | https://orca.cardiff.ac.uk/id/eprint/169623 |
Actions (repository staff only)
Edit Item |