Knowledge distillation for road detection based on cross-model semi-supervised learning

Ma, Wanli, Rosin, Paul L. and Karakus, Oktay 2024. Knowledge distillation for road detection based on cross-model semi-supervised learning. Presented at: IEEE International Geoscience and Remote Sensing Symposium, Athens, Greece, 07-12 July 2024. IGARSS 2024 - 2024 IEEE International Geoscience and Remote Sensing Symposium. IEEE, pp. 8173-8178. 10.1109/IGARSS53475.2024.10641270

Preview

PDF - Accepted Post-Print Version Download (2MB) | Preview

Abstract

The advancement of knowledge distillation has played a crucial role in enabling the transfer of knowledge from larger teacher models to smaller and more efficient student models, and is particularly beneficial for online and resource-constrained applications. The effectiveness of the student model heavily relies on the quality of the distilled knowledge received from the teacher. Given the accessibility of unlabelled remote sensing data, semi-supervised learning has become a prevalent strategy for enhancing model performance. However, relying solely on semi-supervised learning with smaller models may be insufficient due to their limited capacity for feature extraction. This limitation restricts their ability to exploit training data. To address this issue, we propose an integrated approach that combines knowledge distillation and semi-supervised learning methods. This hybrid approach leverages the robust capabilities of large models to effectively utilise large unlabelled data whilst subsequently providing the small student model with rich and informative features for enhancement. The proposed semi-supervised learning-based knowledge distillation (SSLKD) approach demonstrates a notable improvement in the performance of the student model, in the application of road segmentation surpassing the effectiveness of traditional semi-supervised learning methods.

Item Type:	Conference or Workshop Item (Paper)
Date Type:	Published Online
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	IEEE
ISBN:	979-8-3503-6032-5
Date of First Compliant Deposit:	6 November 2024
Last Modified:	20 Nov 2024 16:15
URI:	https://orca.cardiff.ac.uk/id/eprint/173719

Actions (repository staff only)

Edit Item