Animating portrait line drawings from a single face photo and a speech signal

Yi, Ran, Ye, Zipeng, Fan, Ruoyu, Shu, Yezhi, Liu, Yong-Jin, Lai, Yukun ORCID: https://orcid.org/0000-0002-2094-5680 and Rosin, Paul L. ORCID: https://orcid.org/0000-0002-4965-3884 2022. Animating portrait line drawings from a single face photo and a speech signal. Presented at: ACM SIGGRAPH, Vancouver, Canada, 7 - 11 August 2022. SIGGRAPH ’22 Conference Proceedings. ACM, pp. 1-8. 10.1145/3528233.3530720

Preview

PDF - Published Version Available under License Creative Commons Attribution. Download (4MB) | Preview

Abstract

Animating a single face photo is an important research topic which receives considerable attention in computer vision and graphics. Yet line drawings for face portraits, which is a longstanding and popular art form, have not been explored much in this area. Simply concatenating a realistic talking face video generation model with a photo-to-drawing style transfer module suffers from severe inter-frame discontinuity issues. To address this new challenge, we propose a novel framework to generate artistic talking portrait-line-drawing video, given a single face photo and a speech signal. After predicting facial landmark movements from the input speech signal, we propose a novel GAN model to simultaneously handle domain transfer (from photo to drawing) and facial geometry change (according to the predicted facial landmarks). To address the inter-frame discontinuity issues, we propose two novel temporal coherence losses: one based on warping and the other based on a temporal coherence discriminator. Experiments show that our model produces high quality artistic talking portrait-line-drawing videos and outperforms baseline methods. We also show our method can be easily extended to other artistic styles and generate good results. The source code is available at https://github.com/AnimatePortrait/AnimatePortrait .

Item Type:	Conference or Workshop Item (Paper)
Date Type:	Published Online
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	ACM
ISBN:	9781450393379/22/08
Date of First Compliant Deposit:	14 June 2022
Date of Acceptance:	22 April 2022
Last Modified:	27 Oct 2022 15:23
URI:	https://orca.cardiff.ac.uk/id/eprint/150490

Actions (repository staff only)

Edit Item