LMMS reloaded: Transformer-based sense embeddings for disambiguation and beyond

Loureiro, Daniel, Mário Jorge, Alípio and Camacho-Collados, Jose ORCID: https://orcid.org/0000-0003-1618-7239 2022. LMMS reloaded: Transformer-based sense embeddings for disambiguation and beyond. Artificial Intelligence 305 , 103661. 10.1016/j.artint.2022.103661

Preview

PDF - Accepted Post-Print Version Download (3MB) | Preview

Abstract

Distributional semantics based on neural approaches is a cornerstone of Natural Language Processing, with surprising connections to human meaning representation as well. Recent Transformer-based Language Models have proven capable of producing contextual word representations that reliably convey sense-specific information, simply as a product of self-supervision. Prior work has shown that these contextual representations can be used to accurately represent large sense inventories as sense embeddings, to the extent that a distance-based solution to Word Sense Disambiguation (WSD) tasks outperforms models trained specifically for the task. Still, there remains much to understand on how to use these Neural Language Models (NLMs) to produce sense embeddings that can better harness each NLM's meaning representation abilities. In this work we introduce a more principled approach to leverage information from all layers of NLMs, informed by a probing analysis on 14 NLM variants. We also emphasize the versatility of these sense embeddings in contrast to task-specific models, applying them on several sense-related tasks, besides WSD, while demonstrating improved performance using our proposed approach over prior work focused on sense embeddings. Finally, we discuss unexpected findings regarding layer and model performance variations, and potential applications for downstream tasks.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	Elsevier
ISSN:	0004-3702
Date of First Compliant Deposit:	23 March 2022
Date of Acceptance:	3 January 2022
Last Modified:	20 Nov 2024 04:00
URI:	https://orca.cardiff.ac.uk/id/eprint/148612

Actions (repository staff only)

Edit Item