Semantic evaluation of transformer based Language Models: Lexical ambiguity, relational knowledge, and state tracking

Rezaee, Kiamehr 2025. Semantic evaluation of transformer based Language Models: Lexical ambiguity, relational knowledge, and state tracking. PhD Thesis, Cardiff University. Item availability restricted.

	PDF (Cardiff University Electronic Publication Form) - Supplemental Material Restricted to Repository staff only Download (443kB)
Preview	PDF - Accepted Post-Print Version Available under License Creative Commons Attribution Non-commercial No Derivatives. Download (11MB) | Preview

Abstract

Transformer-based Language Models (LMs) have achieved remarkable progress in recent years, yet it remains unclear to what extent their impressive performance reflects genuine understanding or the ability to exploit statistical patterns captured in their pre-training data. This thesis investigates this question through a focused study of semantic behavior and state dependent reasoning in transformer based models. The first part of the thesis examines classic semantic abilities through two complementary phenomena: disambiguation of word senses and relational semantics. We analyze how early and mid generation models encode distinctions in meaning and capture structured relations, and we propose new probing methodologies tailored to these tasks. These include a similarity based prompting approach that enables larger models to handle fine grained disambiguation and an analogy based evaluation framework for probing relational knowledge. To broaden these analyses beyond clean, curated datasets, we extend the study of disambiguation to a noisy real world domain and introduce a task formulation and dataset that allow models to retrieve entity like targets in ambiguous social media text without relying on external knowledge bases. While these semantic tasks shed light on important aspects of model behavior, their finite input spaces make it difficult to determine whether success reflects understanding or memorization. The second part of the thesis therefore turns to dynamic settings and investigates whether transformer based models can update and maintain information as it changes over time. We introduce a suite of state tracking tasks designed to probe behavior beyond surface form recall and evaluate the performance of recent large scale models under this controlled framework. These tasks require models to follow sequences of incremental changes rather than retrieve fixed associations, providing a complementary perspective on their underlying capabilities. Across these studies, the thesis highlights the limitations and strengths of different generations of transformer based models, the importance of appropriate probing methodologies, and the challenges of distinguishing true understanding from sophisticated pattern matching. The results contribute both methodological tools and empirical insights toward a more precise characterization of what current LMs do and do not capture about meaning.

Item Type:	Thesis (PhD)
Date Type:	Completion
Status:	Unpublished
Schools:	Schools > Computer Science & Informatics
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Date of First Compliant Deposit:	18 June 2026
Date of Acceptance:	5 June 2026
Last Modified:	25 Jun 2026 12:04
URI:	https://orca.cardiff.ac.uk/id/eprint/187635

Actions (repository staff only)

Edit Item