HF-MAI: Touchless mid-air interaction for individuals with dexterity impairment

Bamoallem, Razan, Loizides, Fernando ORCID: https://orcid.org/0000-0003-0531-6760, Finnegan, Daniel ORCID: https://orcid.org/0000-0003-1169-2842 and Alsenaidi, Hanin 2025. HF-MAI: Touchless mid-air interaction for individuals with dexterity impairment. Presented at: 20th IFIP TC13 International Conference on Human-Computer Interaction, Belo Horizonte, Minas Gerais, Brazil, 8 - 12 September 2025. Published in: Ardito, C. ed. Proceedings of the Human-Computer Interaction – INTERACT 2025. Lecture Notes in Computer Science , vol.16108 Springer, pp. 108-130. 10.1007/978-3-032-04999-5_7

PDF - Accepted Post-Print Version Download (3MB)

Abstract

For individuals with Multiple Sclerosis (MS) and dexterity impairments (MS-Dex), touchscreen interaction can be particularly challenging, as symptoms like finger numbness and impaired motion regulation negatively impact smartphone usability. This study investigated the usability and accessibility of a mid-air gesture interaction technique for MS-Dex. Touchscreen technologies are ubiquitous, but these individuals often face challenges with touch-based interaction. Touchless input methods, like mid-air gestures, offer a promising alternative for enhancing interaction and communication with touch-based technology. We developed an interactive tool (HF-MAI) using hand and finger mid-air gestures for common touchscreen tasks, using the Leap Motion Controller (LMC) for gesture recognition. MS-Dex individuals, MS therapists, and a usability and user experience (UX) expert participated in the evaluation. We assessed usability, user satisfaction, and accessibility of the HF-MAI system. Results indicate HF-MAI is highly usable and accessible; participants performed gestures successfully and reported high satisfaction. Notably, pinch, spread, and rotate (two-finger/single-finger) mid-air gestures were performed with faster completion times, contrasting with the difficulty of such gestures on touchscreens. The gesture vocabulary, particularly the adaptation to single/two-finger movements, facilitated easier interaction, reducing difficulty for users with limited dexterity. This study demonstrates the potential of mid-air gestures using co-created vocabularies to enhance accessibility for individuals with MS, offering a promising alternative to traditional touch-based input. This technology can significantly improve their access to touchscreens, enhancing independence and quality of life.

Item Type:	Conference or Workshop Item (Paper)
Date Type:	Published Online
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	Springer
ISBN:	978-3-032-04998-8
Date of First Compliant Deposit:	7 August 2025
Last Modified:	30 Sep 2025 09:48
URI:	https://orca.cardiff.ac.uk/id/eprint/180300

Actions (repository staff only)

Edit Item