AMenDeD: Modelling concepts by aligning mentions, definitions and decontextualised embeddings

Gajbhiye, Amit, Bouraoui, Zied, Espinosa-Anke, Luis ORCID: https://orcid.org/0000-0001-6830-9176 and Schockaert, Steven ORCID: https://orcid.org/0000-0002-9256-2881 2024. AMenDeD: Modelling concepts by aligning mentions, definitions and decontextualised embeddings. Presented at: The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), 20-25 May 2024.

Preview

PDF - Accepted Post-Print Version Download (315kB) | Preview

Abstract

Contextualised Language Models (LM) improve on traditional word embeddings by encoding the meaning of words in context. However, such models have also made it possible to learn high-quality decontextualised concept embeddings. Three main strategies for learning such embeddings have thus far been considered: (i) fine-tuning the LM to directly predict concept embeddings from the name of the concept itself, (ii) averaging contextualised representations of mentions of the concept in a corpus, and (iii) encoding definitions of the concept. As these strategies have complementary strengths and weaknesses, we propose to learn a unified embedding space in which all three types of representations can be integrated. We show that this allows us to outperform existing approaches in tasks such as ontology completion, which heavily depends on access to high-quality concept embeddings. We furthermore find that mentions and definitions are well-aligned in the resulting space, enabling tasks such as target sense verification, even without the need for any fine-tuning.

Item Type:	Conference or Workshop Item (Paper)
Status:	In Press
Schools:	Computer Science & Informatics
Date of First Compliant Deposit:	8 May 2024
Date of Acceptance:	20 February 2024
Last Modified:	08 May 2024 14:30
URI:	https://orca.cardiff.ac.uk/id/eprint/168187

Actions (repository staff only)

Edit Item