SHERLOCK: Experimental evaluation of a conversational agent for mobile information tasks

Preece, Alun ORCID: https://orcid.org/0000-0003-0349-9057, Webberley, William, Braines, Dave, Zaroukian, Erin G. and Bakdash, Jonathan Z. 2017. SHERLOCK: Experimental evaluation of a conversational agent for mobile information tasks. IEEE Transactions on Human-Machine Systems 47 (6) , pp. 1017-1028. 10.1109/THMS.2017.2700625

PDF - Published Version Available under License Creative Commons Attribution. Download (815kB)

Abstract

Abstract—Controlled Natural Language (CNL) has great potential to support human-machine interaction (HMI) because it provides an information representation that is both human readable and machine processable.We investigated the effectiveness of a CNL-based conversational interface for HMI in a behavioural experiment called Simple Human Experiment Regarding Locally Observed Collective Knowledge (SHERLOCK). In SHERLOCK, individuals acted in groups to discover and report information to the machine using natural language (NL), which the machine then processed into CNL. The machine fused responses from different users to form a common operating picture, a dashboard showing the level of agreement for distinct information. To obtain information to add to this dashboard, users explored the real world in a simulated crowd-sourced sensing scenario. This scenario represented a simplified, controlled analogue for tactical intelligence (i.e., direct intelligence of the environment), which is key for rapidly planning military, law enforcement, and emergency operations. Overall, despite close to zero training, 74% of the users inputted NL that was machine interpretable and addressed the assigned tasks. An experimental manipulation aimed to increase user-machine interaction, however, did not improve performance as hypothesised. Nevertheless, results indicate the conversational interface may be effective in assisting humans with collection and fusion of information in a crowd-sourcing context.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics Crime and Security Research Institute (CSURI)
Uncontrolled Keywords:	Controlled natural language (CNL), conversational interface, human–computer collaboration (HCC), human–machine interaction (HMI), tactical intelligence Collaboration, Usability, Sensors, Training, Speech, Natural languages, Context
Additional Information:	This work is licensed under a Creative Commons Attribution 3.0 License.
Publisher:	Institute of Electrical and Electronics Engineers (IEEE)
ISSN:	2168-2291
Funders:	U.S. and U.K. Governments
Date of First Compliant Deposit:	26 April 2017
Date of Acceptance:	9 April 2017
Last Modified:	04 May 2023 05:47
URI:	https://orca.cardiff.ac.uk/id/eprint/100131

Citation Data

Cited 12 times in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item