Manifold alignment for semantically aligned style transfer

Huo, Jing, Jin, Shiyin, Li, Wenbin, Wu, Jing ORCID: https://orcid.org/0000-0001-5123-9861, Lai, Yukun ORCID: https://orcid.org/0000-0002-2094-5680, Shi, Yinghuan and Gao, Yang 2021. Manifold alignment for semantically aligned style transfer. Presented at: CVF/IEEE International Conference on Computer Vision (ICCV 2021), Virtual / Montreal, QC, Canada, 11-17 October 2021. Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, pp. 14841-14849. 10.1109/ICCV48922.2021.01459

PDF - Accepted Post-Print Version Download (9MB)

Abstract

Most existing style transfer methods follow the assumption that styles can be represented with global statistics (e.g., Gram matrices or covariance matrices), and thus address the problem by forcing the output and style images to have similar global statistics. An alternative is the assumption of local style patterns, where algorithms are designed to swap similar local features of content and style images. However, the limitation of these existing methods is that they neglect the semantic structure of the content image which may lead to corrupted content structure in the output. In this paper, we make a new assumption that image features from the same semantic region form a manifold and an image with multiple semantic regions follows a multi-manifold distribution. Based on this assumption, the style transfer problem is formulated as aligning two multi-manifold distributions and a Manifold Alignment based Style Transfer (MAST) framework is proposed. The proposed frame-work allows semantically similar regions between the output and the style image share similar style patterns. Moreover, the proposed manifold alignment method is flexible to allow user editing or using semantic segmentation maps as guidance for style transfer. To allow the method to be applicable to photorealistic style transfer, we propose a new adaptive weight skip connection network structure to preserve the content details. Extensive experiments verify the effectiveness of the proposed framework for both artistic and photorealistic style transfer. Code is available at https://github.com/NJUHuoJing/MAST.

Item Type:	Conference or Workshop Item (Paper)
Date Type:	Published Online
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	IEEE
ISBN:	978-1-6654-2813-2
ISSN:	1550-5499
Date of First Compliant Deposit:	2 September 2021
Last Modified:	31 Jul 2025 12:15
URI:	https://orca.cardiff.ac.uk/id/eprint/143848

Citation Data

Cited 9 times in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item