Reference-based deep line art video colorization

Shi, Min, Zhang, Jia-Qi, Chen, Shu-Yu, Gao, Lin, Lai, Yukun ORCID: https://orcid.org/0000-0002-2094-5680 and Zhang, Fang-Lue 2023. Reference-based deep line art video colorization. IEEE Transactions on Visualization and Computer Graphics 29 (6) , pp. 2965-2979. 10.1109/TVCG.2022.3146000

Preview

PDF - Accepted Post-Print Version Download (9MB) | Preview

Abstract

Coloring line art images based on the colors of reference images is an important stage in animation production, which is time-consuming and tedious. In this paper, we propose a deep architecture to automatically color line art videos with the same color style as the given reference images. Our framework consists of a color transform network and a temporal refinement network based on 3U-net. The color transform network takes the target line art images as well as the line art and color images of the reference images as input, and generates corresponding target color images. To cope with the large differences between each target line art image and the reference color images, we propose a distance attention layer that utilizes non-local similarity matching to determine the region correspondences between the target image and the reference images and transforms the local color information from the references to the target. To ensure global color style consistency, we further incorporate Adaptive Instance Normalization (AdaIN) with the transformation parameters obtained from a multiple-layer AdaIN that describes the global color style of the references, extracted by an embedder network. The temporal refinement network learns spatiotemporal features through 3D convolutions to ensure the temporal color consistency of the results. Our model can achieve even better coloring results by fine-tuning the parameters with only a small number of samples when dealing with an animation of a new style. To evaluate our method, we build a line art coloring dataset.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	Institute of Electrical and Electronics Engineers
ISSN:	1077-2626
Funders:	The Royal Society
Date of First Compliant Deposit:	23 January 2022
Date of Acceptance:	7 January 2022
Last Modified:	06 Nov 2023 13:36
URI:	https://orca.cardiff.ac.uk/id/eprint/146848

Actions (repository staff only)

Edit Item