RelBERT: Embedding relations with language models

Ushio, Asahi, Camacho Collados, Jose ORCID: https://orcid.org/0000-0003-1618-7239 and Schockaert, Steven ORCID: https://orcid.org/0000-0002-9256-2881 2025. RelBERT: Embedding relations with language models. Artificial Intelligence 347 , 104359. 10.1016/j.artint.2025.104359

PDF - Published Version Available under License Creative Commons Attribution. Download (3MB)

Abstract

Many applications need access to background knowledge about how different concepts and entities are related. Although Large Language Models (LLM) can address this need to some extent, LLMs are inefficient and difficult to control. As an alternative, we propose to extract relation embeddings from relatively small language models. In particular, we show that masked language models such as RoBERTa can be straightforwardly fine-tuned for this purpose, using only a small amount of training data. The resulting model, which we call RelBERT, captures relational similarity in a surprisingly fine-grained way, allowing us to set a new state-of-the-art in analogy benchmarks. Crucially, RelBERT is capable of modelling relations that go well beyond what the model has seen during training. For instance, we obtained strong results on relations between named entities with a model that was only trained on lexical relations between concepts, and we observed that RelBERT can recognise morphological analogies despite not being trained on such examples. Overall, we find that RelBERT significantly outperforms strategies based on prompting language models that are several orders of magnitude larger, including recent GPT-based models and open source models.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	Elsevier
ISSN:	0004-3702
Date of First Compliant Deposit:	8 July 2025
Date of Acceptance:	7 May 2025
Last Modified:	08 Jul 2025 13:30
URI:	https://orca.cardiff.ac.uk/id/eprint/179566

Actions (repository staff only)

Edit Item