TimeLMs: Diachronic Language Models from Twitter

Loureiro, Daniel, Barbieri, Francesco, Neves, Leonardo, Espinosa-Anke, Luis ORCID: https://orcid.org/0000-0001-6830-9176 and Camacho-collados, Jose 2022. TimeLMs: Diachronic Language Models from Twitter. Presented at: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (pp. 251-260), Dublin, Ireland, 22 - 27 May 2022. Published in: Basile, Valerio, Kozareva, Zornitsa and Stajner, Sanja eds. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp. 251-260. 10.18653/v1/2022.acl-demo.25

Preview

PDF - Accepted Post-Print Version Download (222kB) | Preview

Abstract

Despite its importance, the time variable has been largely neglected in the NLP and language model literature. In this paper, we present TimeLMs, a set of language models specialized on diachronic Twitter data. We show that a continual learning strategy contributes to enhancing Twitter-based language models’ capacity to deal with future and out-of-distribution tweets, while making them competitive with standardized and more monolithic benchmarks. We also perform a number of qualitative analyses showing how they cope with trends and peaks in activity involving specific named entities or concept drift. TimeLMs is available at github.com/cardiffnlp/timelms.

Item Type:	Conference or Workshop Item (Paper)
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	Association for Computational Linguistics
Date of First Compliant Deposit:	17 December 2024
Date of Acceptance:	1 January 2022
Last Modified:	14 Jan 2025 17:38
URI:	https://orca.cardiff.ac.uk/id/eprint/174769

Actions (repository staff only)

Edit Item