A data-centric analysis for efficient semantic knowledge acquisition in word embeddings

Jadhav, Aishwarya, Anderson, Mark, Camacho Collados, Jose ORCID: https://orcid.org/0000-0003-1618-7239 and Pardos, Zachary 2026. A data-centric analysis for efficient semantic knowledge acquisition in word embeddings. Neural Computing and Applications 38 (15) , 606. 10.1007/s00521-026-12342-2

Full text not available from this repository.

Abstract

Distributional models such as word embeddings and language models are trained on ever-larger corpora at substantial computational and environmental cost, yet how they acquire relational semantic knowledge during training, and which training examples drive that acquisition, remains poorly understood. This paper presents a data-centric analysis of semantic knowledge acquisition in word embeddings, focusing on word analogy and semantic similarity. An influence-based framework is proposed that quantifies the contribution of training batches to intrinsic task performance. Influence scores are computed online during training and used to label batches as positively influential, negatively influential, or neutral. Using these labels, learning dynamics and final performance are analysed across Word2Vec, CBOW, FastText, and GloVe, and scores estimated with a lightweight Word2Vec proxy are shown to transfer to larger models. Training on positively influential data consistently outperforms equal-size random data: for Word2Vec, similarity (Pearson) improves by 20% over equal-size random data and 24% over training on the full corpus; fine-tuning RoBERTa on influence-selected data improves similarity correlation by 46% (Pearson) and 30% (Spearman) over equal-size random data and roughly doubles the base model. The results indicate that, for relational semantics, training-data quality outweighs quantity, and that simple proxy models remain a practical, interpretable tool for efficient data selection.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Schools > Computational & Mathematical Sciences Schools > Computer Science & Informatics
Publisher:	Springer
ISSN:	0941-0643
Date of Acceptance:	12 July 2026
Last Modified:	04 Aug 2026 10:30
URI:	https://orca.cardiff.ac.uk/id/eprint/188714

Actions (repository staff only)

Edit Item