Modelling general properties of nouns by selectively averaging contextualised embeddings

Li, Na, Bouraoui, Zied, Camacho Collados, Jose ORCID: https://orcid.org/0000-0003-1618-7239, Espinosa-Anke, Luis ORCID: https://orcid.org/0000-0001-6830-9176, Gu, Qing and Schockaert, Steven ORCID: https://orcid.org/0000-0002-9256-2881 2021. Modelling general properties of nouns by selectively averaging contextualised embeddings. Presented at: 30th International Joint Conference on Artificial Intelligence (IJCAI 2021), Virtual, 21-26 August 2021. Published in: Zhou, Zhi-Hua ed. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, pp. 3850-3856. 10.24963/ijcai.2021/530

Full text not available from this repository.

Abstract

While the success of pre-trained language models has largely eliminated the need for high-quality static word vectors in many NLP applications, static word vectors continue to play an important role in tasks where word meaning needs to be modelled in the absence of linguistic context. In this paper, we explore how the contextualised embeddings predicted by BERT can be used to produce high-quality word vectors for such domains, in particular related to knowledge base completion, where our focus is on capturing the semantic properties of nouns. We find that a simple strategy of averaging the contextualised embeddings of masked word mentions leads to vectors that outperform the static word vectors learned by BERT, as well as those from standard word embedding models, in property induction tasks. We notice in particular that masking target words is critical to achieve this strong performance, as the resulting vectors focus less on idiosyncratic properties and more on general semantic properties. Inspired by this view, we propose a filtering strategy which is aimed at removing the most idiosyncratic mention vectors, allowing us to obtain further performance gains in property induction.

Item Type:	Conference or Workshop Item - published (Paper)
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	International Joint Conferences on Artificial Intelligence Organization
ISBN:	978-0-9992411-9-6
Date of First Compliant Deposit:	4 June 2021
Last Modified:	02 Aug 2026 05:22
URI:	https://orca.cardiff.ac.uk/id/eprint/141727

Actions (repository staff only)

Edit Item