LinkNet: 2D-3D linked multi-modal network for online semantic segmentation of RGB-D videos

Cai, Jun-Xiong, Mu, Tai-Jiang, Lai, Yu-Kun ORCID: https://orcid.org/0000-0002-2094-5680 and Hu, Shi-Min 2021. LinkNet: 2D-3D linked multi-modal network for online semantic segmentation of RGB-D videos. Computers and Graphics 98 , pp. 37-47. 10.1016/j.cag.2021.04.013

Preview

PDF - Accepted Post-Print Version Available under License Creative Commons Attribution Non-commercial No Derivatives. Download (2MB) | Preview

Abstract

This paper proposes LinkNet, a 2D-3D linked multi-modal network served for online semantic segmentation of RGB-D videos, which is essential for real-time applications such as robot navigation. Existing methods for RGB-D semantic segmentation usually work in the regular image domain, which allows efficient processing using convolutional neural networks (CNNs). However, RGB-D videos are captured from a 3D scene, and different frames can contain useful information of the same local region from different views. Working solely in the image domain fails to utilize such crucial information. Our novel approach is based on joint 2D and 3D analysis. The online process is realized simultaneously with 3D scene reconstruction, from which we set up 2D-3D links between continuous RGB-D frames and 3D point cloud. We combine image color and view-insensitive geometric features generated from the 3D point cloud for multi-modal semantic feature learning. Our LinkNet further uses a recurrent neural network (RNN) module to dynamically maintain the hidden semantic states during 3D fusion, and refines the voxel-based labeling results. The experimental results on SceneNet [1] and ScanNet [2] demonstrate that the semantic segmentation results of our framework are stable and effective.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	Elsevier
ISSN:	0097-8493
Date of First Compliant Deposit:	19 June 2021
Date of Acceptance:	13 April 2021
Last Modified:	06 Dec 2024 00:00
URI:	https://orca.cardiff.ac.uk/id/eprint/142000

Citation Data

Cited 2 times in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item