Yi, Ran, Ye, Zipeng, Fan, Ruoyu, Shu, Yezhi, Liu, Yong-Jin, Lai, Yukun  ORCID: https://orcid.org/0000-0002-2094-5680 and Rosin, Paul L. ORCID: https://orcid.org/0000-0002-4965-3884
2022.
Animating portrait line drawings from a single face photo and a speech signal.
Presented at: ACM SIGGRAPH,
Vancouver, Canada,
7 - 11 August 2022.
SIGGRAPH ’22 Conference Proceedings.
ACM,
pp. 1-8.
10.1145/3528233.3530720
|
Preview |
PDF
- Published Version
Available under License Creative Commons Attribution. Download (4MB) | Preview |
Abstract
Animating a single face photo is an important research topic which receives considerable attention in computer vision and graphics. Yet line drawings for face portraits, which is a longstanding and popular art form, have not been explored much in this area. Simply concatenating a realistic talking face video generation model with a photo-to-drawing style transfer module suffers from severe inter-frame discontinuity issues. To address this new challenge, we propose a novel framework to generate artistic talking portrait-line-drawing video, given a single face photo and a speech signal. After predicting facial landmark movements from the input speech signal, we propose a novel GAN model to simultaneously handle domain transfer (from photo to drawing) and facial geometry change (according to the predicted facial landmarks). To address the inter-frame discontinuity issues, we propose two novel temporal coherence losses: one based on warping and the other based on a temporal coherence discriminator. Experiments show that our model produces high quality artistic talking portrait-line-drawing videos and outperforms baseline methods. We also show our method can be easily extended to other artistic styles and generate good results. The source code is available at https://github.com/AnimatePortrait/AnimatePortrait .
| Item Type: | Conference or Workshop Item (Paper) |
|---|---|
| Date Type: | Published Online |
| Status: | Published |
| Schools: | Schools > Computer Science & Informatics |
| Publisher: | ACM |
| ISBN: | 9781450393379/22/08 |
| Date of First Compliant Deposit: | 14 June 2022 |
| Date of Acceptance: | 22 April 2022 |
| Last Modified: | 27 Oct 2022 15:23 |
| URI: | https://orca.cardiff.ac.uk/id/eprint/150490 |
Actions (repository staff only)
 |
Edit Item |
Dimensions
Dimensions