Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

Mutual-cognition for proactive human-robot collaboration: A mixed reality-enabled visual reasoning-based method

Li, Shufei, You, Yingchao, Zheng, Pai, Wang, Xi Vincent and Wang, Lihui 2024. Mutual-cognition for proactive human-robot collaboration: A mixed reality-enabled visual reasoning-based method. IISE Transactions 10.1080/24725854.2024.2313647

[thumbnail of Mutual-cognition for proactive human robot collaboration A mixed reality-enabled visual reasoning-based method.pdf] PDF - Published Version
Available under License Creative Commons Attribution.

Download (3MB)


Human-Robot Collaboration (HRC) is key to achieving the flexible automation required by the mass personalization trend, especially towards human-centric intelligent manufacturing. Nevertheless, existing HRC systems suffer from poor task understanding and poor ergonomic satisfaction, which impede empathetic teamwork skills in task execution. To overcome the bottleneck, a Mixed Reality (MR) and visual reasoning-based method is proposed in this research, providing mutual-cognitive task assignment for human and robotic agents’ operations. Firstly, an MR-enabled mutual-cognitive HRC architecture is proposed, with the characteristic of monitoring Digital Twins states, reasoning co-working strategies, and providing cognitive services. Secondly, a visual reasoning approach is introduced, which learns scene interpretation from the visual perception of each agent’s actions and environmental changes to make task planning strategies satisfying human–robot operation needs. Lastly, a safe, ergonomic, and proactive robot motion planning algorithm is proposed to let a robot execute generated co-working strategies, while a human operator is supported with intuitive task operation guidance in the MR environment, achieving empathetic collaboration. Through a demonstration of a disassembly task of aging Electric Vehicle Batteries, the experimental result facilitates cognitive intelligence in Proactive HRC for flexible automation.

Item Type: Article
Date Type: Published Online
Status: In Press
Schools: Engineering
Publisher: Taylor and Francis Group
ISSN: 2472-5854
Date of First Compliant Deposit: 19 April 2024
Date of Acceptance: 22 January 2024
Last Modified: 19 Apr 2024 15:15

Actions (repository staff only)

Edit Item Edit Item


Downloads per month over past year

View more statistics