PISE-V: person image and video synthesis with decoupled GAN

Zhang, Jinsong, Lai, Yu-Kun ORCID: https://orcid.org/0000-0002-2094-5680, Yang, Jingyu and Li, Kun 2024. PISE-V: person image and video synthesis with decoupled GAN. Visual Computer 10.1007/s00371-024-03752-1 Item availability restricted.

PDF - Accepted Post-Print Version Restricted to Repository staff only until 15 December 2025 due to copyright restrictions. Download (6MB)

Abstract

Pose-guided image and video synthesis is very challenging due to large variation and occlusion. Failing to disentangle the shape and the style of clothing, previous methods cannot fully control the image generation process, which limits their applications on person image/video editing. In this paper, we design a novel decoupled generator for person image and video synthesis, which is able to generate realistic images with desired poses, textures and semantic layouts. We adopt a two-stage framework with a parsing generator and an image generator to tackle this ill-posed problem. We first synthesize a human semantic parsing aligned with the target pose and then transfer the image information to generate the target image. To decouple the shape and the style of clothing, we propose joint global and local per-region encoding and normalization to predict the reasonable style of clothing for invisible regions. We also propose spatial-aware normalization to retain the spatial details in the source image. In order to capture large deformations for person video synthesis, we propose a region-based flow module with positional encoding to predict the dense correspondences between the source image and the target image, while disentangling the shape and the style of clothing. Experimental results show the superior performance of our model on pose-guided image/video synthesis. Besides, the results of texture transfer and parsing editing show that our model can be applied to person image and video editing. Code, dataset and models are available at: https://github.com/Zhangjinso/PISE.

Item Type:	Article
Date Type:	Published Online
Status:	In Press
Schools:	Computer Science & Informatics
Publisher:	Springer
ISSN:	0178-2789
Date of First Compliant Deposit:	8 February 2025
Date of Acceptance:	3 December 2024
Last Modified:	11 Feb 2025 11:15
URI:	https://orca.cardiff.ac.uk/id/eprint/176048

Actions (repository staff only)

Edit Item