STATE: Learning structure and texture representations for novel view synthesis

Jing, Xinyi, Feng, Qiao, Lai, Yu-Kun ORCID: https://orcid.org/0000-0002-2094-5680, Zhang, Jinsong, Yu, Yuanqiang and Li, Kun 2023. STATE: Learning structure and texture representations for novel view synthesis. Computational Visual Media 9 (4) , pp. 767-786. 10.1007/s41095-022-0301-9

PDF - Published Version Download (10MB)

Abstract

Novel viewpoint image synthesis is very challenging, especially from sparse views, due to large changes in viewpoint and occlusion. Existing image-based methods fail to generate reasonable results for invisible regions, while geometry-based methods have difficulties in synthesizing detailed textures. In this paper, we propose STATE, an end-to-end deep neural network, for sparse view synthesis by learning structure and texture representations. Structure is encoded as a hybrid feature field to predict reasonable structures for invisible regions while maintaining original structures for visible regions, and texture is encoded as a deformed feature map to preserve detailed textures. We propose a hierarchical fusion scheme with intra-branch and inter-branch aggregation, in which spatio-view attention allows multi-view fusion at the feature level to adaptively select important information by regressing pixel-wise or voxel-wise confidence maps. By decoding the aggregated features, STATE is able to generate realistic images with reasonable structures and detailed textures. Experimental results demonstrate that our method achieves qualitatively and quantitatively better results than state-of-the-art methods. Our method also enables texture and structure editing applications benefiting from implicit disentanglement of structure and texture. Our code is available at http://cic.tju.edu.cn/faculty/likun/projects/STATE.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Additional Information:	License information from Publisher: LICENSE 1: URL: http://creativecommons.org/licenses/by/4.0/, Type: open-access
Publisher:	SpringerOpen
ISSN:	2096-0433
Date of First Compliant Deposit:	21 August 2023
Date of Acceptance:	16 June 2022
Last Modified:	22 Aug 2023 07:46
URI:	https://orca.cardiff.ac.uk/id/eprint/161959

Actions (repository staff only)

Edit Item