Li, Bo, Wei, Xiaolin, Liu, Bin, He, Zhifen, Cao, Junjie and Lai, Yu-Kun ORCID: https://orcid.org/0000-0002-2094-5680 2024. Pose-aware 3D talking face synthesis using geometry-guided audio-vertices attention. IEEE Transactions on Visualization and Computer Graphics 10.1109/TVCG.2024.3371064 |
Preview |
PDF
- Accepted Post-Print Version
Download (20MB) | Preview |
Abstract
Most of the existing 3D talking face synthesis methods suffer from the lack of detailed facial expressions and realistic head poses, resulting in unsatisfactory experiences for users. In this paper, we propose a novel pose-aware 3D talking face synthesis method with a novel geometry-guided audio-vertices attention. To capture more detailed expression, such as the subtle nuances of mouth shape and eye movement, we propose to build hierarchical audio features including a global attribute feature and a series of vertex-wise local latent movement features. Then, in order to fully exploit the topology of facial models, we further propose a novel geometry-guided audio-vertices attention module to predict the displacement of each vertex by using vertex connectivity relations to take full advantage of the corresponding hierarchical audio features. Finally, to accomplish pose-aware animation, we expand the existing database with an additional pose attribute, and a novel pose estimation module is proposed by paying attention to the whole head model. Numerical experiments demonstrate the effectiveness of the proposed method on realistic expression and head movements against state-of-the-art methods.
Item Type: | Article |
---|---|
Date Type: | Published Online |
Status: | In Press |
Schools: | Computer Science & Informatics |
Publisher: | Institute of Electrical and Electronics Engineers |
ISSN: | 1077-2626 |
Date of First Compliant Deposit: | 21 March 2024 |
Date of Acceptance: | 20 February 2024 |
Last Modified: | 27 Mar 2024 09:48 |
URI: | https://orca.cardiff.ac.uk/id/eprint/167426 |
Actions (repository staff only)
Edit Item |