Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

Animating portrait line drawings from a single face photo and a speech signal

Yi, Ran, Ye, Zipeng, Fan, Ruoyu, Shu, Yezhi, Liu, Yong-Jin, Lai, Yukun ORCID: and Rosin, Paul L. ORCID: 2022. Animating portrait line drawings from a single face photo and a speech signal. Presented at: ACM SIGGRAPH, Vancouver, Canada, 7 - 11 August 2022. SIGGRAPH ’22 Conference Proceedings. ACM, pp. 1-8. 10.1145/3528233.3530720

[thumbnail of 3528233.3530720.pdf]
PDF - Published Version
Available under License Creative Commons Attribution.

Download (4MB) | Preview


Animating a single face photo is an important research topic which receives considerable attention in computer vision and graphics. Yet line drawings for face portraits, which is a longstanding and popular art form, have not been explored much in this area. Simply concatenating a realistic talking face video generation model with a photo-to-drawing style transfer module suffers from severe inter-frame discontinuity issues. To address this new challenge, we propose a novel framework to generate artistic talking portrait-line-drawing video, given a single face photo and a speech signal. After predicting facial landmark movements from the input speech signal, we propose a novel GAN model to simultaneously handle domain transfer (from photo to drawing) and facial geometry change (according to the predicted facial landmarks). To address the inter-frame discontinuity issues, we propose two novel temporal coherence losses: one based on warping and the other based on a temporal coherence discriminator. Experiments show that our model produces high quality artistic talking portrait-line-drawing videos and outperforms baseline methods. We also show our method can be easily extended to other artistic styles and generate good results. The source code is available at .

Item Type: Conference or Workshop Item (Paper)
Date Type: Published Online
Status: Published
Schools: Computer Science & Informatics
Publisher: ACM
ISBN: 9781450393379/22/08
Date of First Compliant Deposit: 14 June 2022
Date of Acceptance: 22 April 2022
Last Modified: 27 Oct 2022 15:23

Actions (repository staff only)

Edit Item Edit Item


Downloads per month over past year

View more statistics