Learning to reconstruct and understand indoor scenes from sparse views

Yang, Jingyu, Xu, Ji, Li, Kun, Lai, Yukun ORCID: https://orcid.org/0000-0002-2094-5680, Yue, Huanjing, Lu, Jianzhi, Wu, Hao and Liu, Yebin 2020. Learning to reconstruct and understand indoor scenes from sparse views. IEEE Transactions on Image Processing 29 , 5753 -5766. 10.1109/TIP.2020.2986712

Preview

PDF - Accepted Post-Print Version Download (17MB) | Preview

Abstract

This paper proposes a new method for simultaneous 3D reconstruction and semantic segmentation for indoor scenes. Unlike existing methods that require recording a video using a color camera and/or a depth camera, our method only needs a small number of (e.g., 3~5) color images from uncalibrated sparse views, which significantly simplifies data acquisition and broadens applicable scenarios. To achieve promising 3D reconstruction from sparse views with limited overlap, our method first recovers the depth map and semantic information for each view, and then fuses the depth maps into a 3D scene. To this end, we design an iterative deep architecture, named IterNet, to estimate the depth map and semantic segmentation alternately. To obtain accurate alignment between views with limited overlap, we further propose a joint global and local registration method to reconstruct a 3D scene with semantic information. We also make available a new indoor synthetic dataset, containing photorealistic high-resolution RGB images, accurate depth maps and pixel-level semantic labels for thousands of complex layouts. Experimental results on public datasets and our dataset demonstrate that our method achieves more accurate depth estimation, smaller semantic segmentation errors, and better 3D reconstruction results over state-of-the-art methods.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	Institute of Electrical and Electronics Engineers (IEEE)
ISSN:	1057-7149
Date of First Compliant Deposit:	6 April 2020
Date of Acceptance:	28 March 2020
Last Modified:	27 Nov 2024 03:45
URI:	https://orca.cardiff.ac.uk/id/eprint/130860

Citation Data

Cited 1 time in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item