Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

Deep line art video colorization with a few references

Shi, Min, Zhang, Jia-Qi, Chen, Shu-Yu, Gao, Lin, Lai, Yukun ORCID: https://orcid.org/0000-0002-2094-5680 and Zhang, Fang-Lue 2022. Deep line art video colorization with a few references. IEEE Transactions on Visualization and Computer Graphics 10.1109/TVCG.2022.3146000

[thumbnail of RefLineArtVideoColorizationTVCG.pdf]
Preview
PDF - Accepted Post-Print Version
Download (9MB) | Preview

Abstract

Coloring line art images based on the colors of reference images is an important stage in animation production, which is time-consuming and tedious. In this paper, we propose a deep architecture to automatically color line art videos with the same color style as the given reference images. Our framework consists of a color transform network and a temporal refinement network based on 3U-net. The color transform network takes the target line art images as well as the line art and color images of the reference images as input, and generates corresponding target color images. To cope with the large differences between each target line art image and the reference color images, we propose a distance attention layer that utilizes non-local similarity matching to determine the region correspondences between the target image and the reference images and transforms the local color information from the references to the target. To ensure global color style consistency, we further incorporate Adaptive Instance Normalization (AdaIN) with the transformation parameters obtained from a multiple-layer AdaIN that describes the global color style of the references, extracted by an embedder network. The temporal refinement network learns spatiotemporal features through 3D convolutions to ensure the temporal color consistency of the results. Our model can achieve even better coloring results by fine-tuning the parameters with only a small number of samples when dealing with an animation of a new style. To evaluate our method, we build a line art coloring dataset.

Item Type: Article
Date Type: Published Online
Status: In Press
Schools: Computer Science & Informatics
Additional Information: Title slightly changed on publication
Publisher: Institute of Electrical and Electronics Engineers
ISSN: 1077-2626
Funders: The Royal Society
Date of First Compliant Deposit: 23 January 2022
Date of Acceptance: 7 January 2022
Last Modified: 10 Nov 2022 11:24
URI: https://orca.cardiff.ac.uk/id/eprint/146848

Actions (repository staff only)

Edit Item Edit Item

Downloads

Downloads per month over past year

View more statistics