Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

Pose-aware 3D talking face synthesis using geometry-guided audio-vertices attention

Li, Bo, Wei, Xiaolin, Liu, Bin, He, Zhifen, Cao, Junjie and Lai, Yu-Kun ORCID: 2024. Pose-aware 3D talking face synthesis using geometry-guided audio-vertices attention. IEEE Transactions on Visualization and Computer Graphics 10.1109/TVCG.2024.3371064

[thumbnail of PoseAware3DFacialAnimationSynthesisTVCG.pdf]
PDF - Accepted Post-Print Version
Download (20MB) | Preview


Most of the existing 3D talking face synthesis methods suffer from the lack of detailed facial expressions and realistic head poses, resulting in unsatisfactory experiences for users. In this paper, we propose a novel pose-aware 3D talking face synthesis method with a novel geometry-guided audio-vertices attention. To capture more detailed expression, such as the subtle nuances of mouth shape and eye movement, we propose to build hierarchical audio features including a global attribute feature and a series of vertex-wise local latent movement features. Then, in order to fully exploit the topology of facial models, we further propose a novel geometry-guided audio-vertices attention module to predict the displacement of each vertex by using vertex connectivity relations to take full advantage of the corresponding hierarchical audio features. Finally, to accomplish pose-aware animation, we expand the existing database with an additional pose attribute, and a novel pose estimation module is proposed by paying attention to the whole head model. Numerical experiments demonstrate the effectiveness of the proposed method on realistic expression and head movements against state-of-the-art methods.

Item Type: Article
Date Type: Published Online
Status: In Press
Schools: Computer Science & Informatics
Publisher: Institute of Electrical and Electronics Engineers
ISSN: 1077-2626
Date of First Compliant Deposit: 21 March 2024
Date of Acceptance: 20 February 2024
Last Modified: 27 Mar 2024 09:48

Actions (repository staff only)

Edit Item Edit Item


Downloads per month over past year

View more statistics