Efficient hierarchical reinforcement learning for mapless navigation with predictive neighbouring space scoring

Gao, Yan, Wu, Jing, Yang, Xintong ORCID: https://orcid.org/0000-0002-7612-614X and Ji, Ze ORCID: https://orcid.org/0000-0002-8968-9902 2023. Efficient hierarchical reinforcement learning for mapless navigation with predictive neighbouring space scoring. IEEE Transactions on Automation Science and Engineering 10.1109/TASE.2023.3312237

PDF - Accepted Post-Print Version Download (4MB)

Abstract

Solving reinforcement learning (RL)-based mapless navigation tasks is challenging due to their sparse reward and long decision horizon nature. Hierarchical reinforcement learning (HRL) has the ability to leverage knowledge at different abstract levels and is thus preferred in complex mapless navigation tasks. However, it is computationally expensive and inefficient to learn navigation end-to-end from raw high-dimensional sensor data, such as Lidar or RGB cameras. The use of subgoals based on a compact intermediate representation is therefore preferred for dimension reduction. This work proposes an efficient HRL-based framework to achieve this with a novel scoring method, named Predictive Neighbouring Space Scoring (PNSS). The PNSS model estimates the explorable space for a given position of interest based on the current robot observation. The PNSS values for a few candidate positions around the robot provide a compact and informative state representation for subgoal selection. We study the effects of different candidate position layouts and demonstrate that our layout design facilitates higher performances in longer-range tasks. Moreover, a penalty term is introduced in the reward function for the high-level (HL) policy, so that the subgoal selection process takes the performance of the low-level (LL) policy into consideration. Comprehensive evaluations demonstrate that using the proposed PNSS module consistently improves performances over the use of Lidar only or Lidar and encoded RGB features.

Item Type:	Article
Date Type:	Published Online
Status:	In Press
Schools:	Engineering
Publisher:	Institute of Electrical and Electronics Engineers
ISSN:	1545-5955
Date of First Compliant Deposit:	5 September 2023
Date of Acceptance:	30 August 2023
Last Modified:	09 Nov 2023 10:19
URI:	https://orca.cardiff.ac.uk/id/eprint/162242

Actions (repository staff only)

Edit Item