Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

Animating portrait line drawings from a single face photo and a speech signal

Yi, Ran, Ye, Zipeng, Fan, Ruoyu, Shu, Yezhi, Liu, Yong-Jin, Lai, Yukun ORCID: https://orcid.org/0000-0002-2094-5680 and Rosin, Paul L. ORCID: https://orcid.org/0000-0002-4965-3884 2022. Animating portrait line drawings from a single face photo and a speech signal. Presented at: ACM SIGGRAPH, Vancouver, Canada, 7 - 11 August 2022. SIGGRAPH ’22 Conference Proceedings. ACM, pp. 1-8. 10.1145/3528233.3530720

[thumbnail of 3528233.3530720.pdf]
Preview
PDF - Published Version
Available under License Creative Commons Attribution.

Download (4MB) | Preview

Abstract

Animating a single face photo is an important research topic which receives considerable attention in computer vision and graphics. Yet line drawings for face portraits, which is a longstanding and popular art form, have not been explored much in this area. Simply concatenating a realistic talking face video generation model with a photo-to-drawing style transfer module suffers from severe inter-frame discontinuity issues. To address this new challenge, we propose a novel framework to generate artistic talking portrait-line-drawing video, given a single face photo and a speech signal. After predicting facial landmark movements from the input speech signal, we propose a novel GAN model to simultaneously handle domain transfer (from photo to drawing) and facial geometry change (according to the predicted facial landmarks). To address the inter-frame discontinuity issues, we propose two novel temporal coherence losses: one based on warping and the other based on a temporal coherence discriminator. Experiments show that our model produces high quality artistic talking portrait-line-drawing videos and outperforms baseline methods. We also show our method can be easily extended to other artistic styles and generate good results. The source code is available at https://github.com/AnimatePortrait/AnimatePortrait .

Item Type: Conference or Workshop Item (Paper)
Date Type: Published Online
Status: Published
Schools: Computer Science & Informatics
Publisher: ACM
ISBN: 9781450393379/22/08
Date of First Compliant Deposit: 14 June 2022
Date of Acceptance: 22 April 2022
Last Modified: 27 Oct 2022 15:23
URI: https://orca.cardiff.ac.uk/id/eprint/150490

Actions (repository staff only)

Edit Item Edit Item

Downloads

Downloads per month over past year

View more statistics