Coarse- and fine-grained fusion hierarchical network for hole filling in view synthesis

Wang, Guangcheng, Jiang, Kui, Gu, Ke, Liu, Hongyan, Liu, Hantao ORCID: https://orcid.org/0000-0003-4544-3481 and Zhang, Wenjun 2024. Coarse- and fine-grained fusion hierarchical network for hole filling in view synthesis. IEEE Transactions on Image Processing 33 , pp. 322-337. 10.1109/TIP.2023.3341303

PDF - Accepted Post-Print Version Download (29MB)

Abstract

Depth image-based rendering (DIBR) techniques play an essential role in free-viewpoint videos (FVVs), which generate the virtual views from a reference 2D texture video and its associated depth information. However, the background regions occluded by the foreground in the reference view will be exposed in the synthesized view, resulting in obvious irregular holes in the synthesized view. To this end, this paper proposes a novel coarse and fine-grained fusion hierarchical network (CFFHNet) for hole filling, which fills the irregular holes produced by view synthesis using the spatial contextual correlations between the visible and hole regions. CFFHNet adopts recurrent calculation to learn the spatial contextual correlation, while the hierarchical structure and attention mechanism are introduced to guide the fine-grained fusion of cross-scale contextual features. To promote texture generation while maintaining fidelity, we equip CFFHNet with a two-stage framework involving an inference sub-network to generate the coarse synthetic result and a refinement sub-network for refinement. Meanwhile, to make the learned hole-filling model better adaptable and robust to the “foreground penetration” distortion, we trained CFFHNet by generating a batch of training samples by adding irregular holes to the foreground and background connection regions of high-quality images. Extensive experiments show the superiority of our CFFHNet over the current state-of-the-art DIBR methods. The source code will be available at https://github.com/wgc-vsfm/view-synthesis-CFFHNet .

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	Institute of Electrical and Electronics Engineers
ISSN:	1057-7149
Date of First Compliant Deposit:	23 December 2023
Date of Acceptance:	7 December 2023
Last Modified:	19 Feb 2024 16:48
URI:	https://orca.cardiff.ac.uk/id/eprint/165048

Actions (repository staff only)

Edit Item