Deep reinforcement learning with explicit context representation

Munguia-Galeano, Francisco ORCID: https://orcid.org/0000-0001-8397-3083, Tan, Ah-Hwee and Ji, Ze ORCID: https://orcid.org/0000-0002-8968-9902 2023. Deep reinforcement learning with explicit context representation. IEEE Transactions on Neural Networks and Learning Systems 10.1109/TNNLS.2023.3325633

PDF - Accepted Post-Print Version Download (7MB)

Abstract

Though Reinforcement learning (RL) has shown an outstanding capability for solving complex computational problems, most RL algorithms lack an explicit method that would allow learning from contextual information. On the other hand, humans often use context to identify patterns and relations among elements in the environment, along with how to avoid making wrong actions. However, what may seem like an obviously wrong decision from a human perspective could take hundreds of steps for an RL agent to learn to avoid. This paper proposes a framework for discrete environments called Iota explicit context representation (IECR). The framework involves representing each state using contextual key frames (CKFs), which can then be used to extract a function that represents the affordances of the state; in addition, two loss functions are introduced with respect to the affordances of the state. The novelty of the IECR framework lies in its capacity to extract contextual information from the environment and learn from the CKFs’ representation. We validate the framework by developing four new algorithms that learn using context: Iota deep Q-network (IDQN), Iota double deep Q-network (IDDQN), Iota dueling deep Q-network (IDuDQN), and Iota dueling double deep Q-network (IDDDQN). Furthermore, we evaluate the framework and the new algorithms in five discrete environments. We show that all the algorithms, which use contextual information, converge in around 40,000 training steps of the neural networks, significantly outperforming their state-of-the-art equivalents.

Item Type:	Article
Date Type:	Published Online
Status:	In Press
Schools:	Engineering
Publisher:	Institute of Electrical and Electronics Engineers
ISSN:	2162-237X
Date of First Compliant Deposit:	17 October 2023
Date of Acceptance:	15 October 2023
Last Modified:	01 Jun 2024 14:23
URI:	https://orca.cardiff.ac.uk/id/eprint/163231

Citation Data

Cited 1 time in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item