Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

RelBERT: Embedding relations with language models

Ushio, Asahi, Camacho Collados, Jose ORCID: https://orcid.org/0000-0003-1618-7239 and Schockaert, Steven ORCID: https://orcid.org/0000-0002-9256-2881 2025. RelBERT: Embedding relations with language models. Artificial Intelligence 347 , 104359. 10.1016/j.artint.2025.104359

[thumbnail of 1-s2.0-S0004370225000785-main.pdf] PDF - Published Version
Available under License Creative Commons Attribution.

Download (3MB)

Abstract

Many applications need access to background knowledge about how different concepts and entities are related. Although Large Language Models (LLM) can address this need to some extent, LLMs are inefficient and difficult to control. As an alternative, we propose to extract relation embeddings from relatively small language models. In particular, we show that masked language models such as RoBERTa can be straightforwardly fine-tuned for this purpose, using only a small amount of training data. The resulting model, which we call RelBERT, captures relational similarity in a surprisingly fine-grained way, allowing us to set a new state-of-the-art in analogy benchmarks. Crucially, RelBERT is capable of modelling relations that go well beyond what the model has seen during training. For instance, we obtained strong results on relations between named entities with a model that was only trained on lexical relations between concepts, and we observed that RelBERT can recognise morphological analogies despite not being trained on such examples. Overall, we find that RelBERT significantly outperforms strategies based on prompting language models that are several orders of magnitude larger, including recent GPT-based models and open source models.

Item Type: Article
Date Type: Publication
Status: Published
Schools: Schools > Computer Science & Informatics
Publisher: Elsevier
ISSN: 0004-3702
Date of First Compliant Deposit: 8 July 2025
Date of Acceptance: 7 May 2025
Last Modified: 08 Jul 2025 13:30
URI: https://orca.cardiff.ac.uk/id/eprint/179566

Actions (repository staff only)

Edit Item Edit Item

Downloads

Downloads per month over past year

View more statistics