Learning dynamic contextualised word embeddings via template-based temporal adaptation

Tang, Xiaohang, Zhou, Yi ORCID: https://orcid.org/0000-0001-7009-8515 and Bollegala, Danushka 2023. Learning dynamic contextualised word embeddings via template-based temporal adaptation. Presented at: 61st Annual Meeting of the Association of Computational Linguistics, Toronto, Canada, 9 - 14 July 2023. Published in: Rogers, Anna, Boyd-Graber, Jordan and Okazaki, Naoaki eds. Proceedings of the 61st Meeting of the Association of Computational Linguistics (Volume 1: Long Papers). Association of Computational Linguistics, 10.18653/v1/2023.acl-long.520

Full text not available from this repository.

Abstract

Dynamic contextualised word embeddings (DCWEs) represent the temporal semantic variations of words. We propose a method for learning DCWEs by time-adapting a pretrained Masked Language Model (MLM) using time-sensitive templates. Given two snapshots C1 and C2 of a corpus taken respectively at two distinct timestamps T1 and T2, we first propose an unsupervised method to select (a) pivot terms related to both C1 and C2, and (b) anchor terms that are associated with a specific pivot term in each individual snapshot.We then generate prompts by filling manually compiled templates using the extracted pivot and anchor terms.Moreover, we propose an automatic method to learn time-sensitive templates from C1 and C2, without requiring any human supervision.Next, we use the generated prompts to adapt a pretrained MLM to T2 by fine-tuning using those prompts.Multiple experiments show that our proposed method significantly reduces the perplexity of test sentences in C2, outperforming the current state-of-the-art.

Item Type:	Conference or Workshop Item (Paper)
Date Type:	Published Online
Status:	Unpublished
Schools:	Computer Science & Informatics
Publisher:	Association of Computational Linguistics
Last Modified:	01 Aug 2024 14:30
URI:	https://orca.cardiff.ac.uk/id/eprint/170399

Actions (repository staff only)

Edit Item