Real-time 3D face reconstruction and gaze tracking for virtual reality

Chen, Shu-Yu, Gao, Lin, Lai, Yukun ORCID: https://orcid.org/0000-0002-2094-5680, Rosin, Paul ORCID: https://orcid.org/0000-0002-4965-3884 and Xia, Shihong 2018. Real-time 3D face reconstruction and gaze tracking for virtual reality. Presented at: IEEE Conference on Virtual Reality and 3D User Interfaces, Reutlingen, Germany, 18-22 March 2018. -.

Preview

PDF - Presentation Download (3MB) | Preview

Abstract

With the rapid development of virtual reality (VR) technology, VR glasses, a.k.a. Head-Mounted Displays (HMDs) are widely available, allowing immersive 3D content to be viewed. A natural need for truly immersive VR is to allow bidirectional communication: the user should be able to interact with the virtual world using facial expressions and eye gaze, in addition to traditional means of interaction. Typical application scenarios include VR virtual conferencing and virtual roaming, where ideally users are able to see other users’ expressions and have eye contact with them in the virtual world. Despite significant achievements in recent years for reconstruction of 3D faces from RGB or RGB-D images, it remains a challenge to reliably capture and reconstruct 3D facial expressions including eye gaze when the user is wearing VR glasses, because the majority of the face is occluded, especially those areas around the eyes which are essential for recognizing facial expressions and eye gaze. In this paper, we introduce a novel real-time system that is able to capture and reconstruct 3D faces wearing HMDs and robustly recover eye gaze. We demonstrate the effectiveness of our system using live capture and more results are shown in the accompanying video.

Item Type:	Conference or Workshop Item (Paper)
Status:	Unpublished
Schools:	Computer Science & Informatics
Funders:	Royal Society
Date of First Compliant Deposit:	7 March 2018
Last Modified:	23 Oct 2022 13:07
URI:	https://orca.cardiff.ac.uk/id/eprint/109716

Citation Data

Cited 8 times in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item