Hierarchical reinforcement learning-based mapless navigation with predictive exploration worthiness

Gao, Yan, Ji, Ze ORCID: https://orcid.org/0000-0002-8968-9902, Wu, Jing ORCID: https://orcid.org/0000-0001-5123-9861, Wei, Changyun and Grech, Raphael 2023. Hierarchical reinforcement learning-based mapless navigation with predictive exploration worthiness. Presented at: IEEE International Conference on Mechatronics and Automation, Harbin, Heilongjiang, China, 6-9 August 2023. Proceedings of the 2023 IEEE International Conference on Mechatronics and Automation. IEEE, pp. 636-643. 10.1109/ICMA57826.2023.10215569

PDF - Accepted Post-Print Version Download (3MB)

Abstract

Hierarchical reinforcement learning (HRL) is a promising approach for complex mapless navigation tasks by decomposing the task into a hierarchy of subtasks. However, selecting appropriate subgoals is challenging. Existing methods predominantly rely on sensory inputs, which may contain inadequate information or excessive redundancy. Inspired by the cognitive processes underpinning human navigation, our aim is to enable the robot to leverage both ‘intrinsic and extrinsic factors’ to make informed decisions regarding subgoal selection. In this work, we propose a novel HRL-based mapless navigation framework. Specifically, we introduce a predictive module, named Predictive Exploration Worthiness (PEW), into the high-level (HL) decision-making policy. The hypothesis is that the worthiness of an area for further exploration is related to obstacle spatial distribution, such as the area of free space and the distribution of obstacles. The PEW is introduced as a compact representation for obstacle spatial distribution. Additionally, to incorporate ‘intrinsic factors’ in the subgoal selection process, a penalty element is introduced in the HL reward function, allowing the robot to take into account the capabilities of the low-level policy when selecting subgoals. Our method exhibits significant improvements in success rate when tested in unseen environments.

Item Type:	Conference or Workshop Item - published (Paper)
Date Type:	Published Online
Status:	Published
Schools:	Schools > Engineering Schools > Computer Science & Informatics
Publisher:	IEEE
ISBN:	979835032085-
ISSN:	2152-7431
Date of First Compliant Deposit:	16 May 2023
Date of Acceptance:	15 May 2023
Last Modified:	31 Mar 2026 14:19
URI:	https://orca.cardiff.ac.uk/id/eprint/159543

Actions (repository staff only)

Edit Item