Probing the conceptual space of ChatGPT and GPT-4

Chatterjee, Usashi and Schockaert, Steven ORCID: https://orcid.org/0000-0002-9256-2881 2023. Probing the conceptual space of ChatGPT and GPT-4. Presented at: 9th Workshop on Artificial Intelligence and Cognition, Bremen, 14-15 September 2023.

Preview

PDF - Published Version Available under License Creative Commons Attribution. Download (202kB) | Preview

Abstract

Distilling knowledge from Large Language Models (LLMs) has emerged as a promising strategy for populating knowledge bases with factual knowledge. The aim of this paper is to explore the feasibility of similarly using LLMs for learning cognitively plausible representations of concepts, focusing in particular on the framework of conceptual spaces. Such representations allow us to compare concepts along particular quality dimensions, e.g. in terms of their size, colour or shape. Learning conceptual spaces is known to be challenging, among others because many of the features that need to be captured are rarely expressed in text (e.g. shape), a problem which is exacerbated by reporting bias. In this paper, we explore to what extent recent LLMs are able to overcome these barriers. To this end, we introduce a new dataset with three types of probing questions. Our results provide evidence that ChatGPT has access to a rich conceptual structure, which allows it to make connections between unrelated concepts (e.g. the fact that limousines and crocodiles have a similar shape). On the other hand, we also find that the model sometimes falls back on shallow heuristics. Compared to ChatGPT, GPT-4 makes fewer mistakes, although the difference in performance is generally small.

Item Type:	Conference or Workshop Item (Paper)
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Date of First Compliant Deposit:	13 February 2024
Date of Acceptance:	4 August 2023
Last Modified:	15 Feb 2024 10:29
URI:	https://orca.cardiff.ac.uk/id/eprint/165641

Actions (repository staff only)

Edit Item