Representing relational knowledge with language models

Ushio, Asahi 2024. Representing relational knowledge with language models. PhD Thesis, Cardiff University. Item availability restricted.

Preview	PDF (Asahi Ushio, PhD Thesis) - Accepted Post-Print Version Available under License Creative Commons Attribution Non-commercial No Derivatives. Download (9MB) | Preview
	PDF (Cardiff University Electronic Theses and Dissertations Form - Memorandum of Understanding) - Supplemental Material Restricted to Repository staff only Download (203kB)

Abstract

Relational knowledge is the ability to recognize the relationship between instances, and it has an important role in human understanding a concept or commonsense reasoning. We, humans, structure our knowledge by understanding individual instances together with the relationship among them, which enables us to further expand the knowledge. Nevertheless, modelling relational knowledge with computational models is a long-standing challenge in Natural Language Processing (NLP). The main difficulty at acquiring relational knowledge arises from the generalization capability. For pre-trained Language Model (LM), in spite of the huge impact made in NLP, relational knowledge remains understudied. In fact, GPT-3 (Brown et al., 2020), one of the largest LM at the time being with 175 billions of parameters, has shown to perform worse than a traditional statistical baseline in an analogy benchmark. Our initial results hinted at the type of relational knowledge encoded in some of the LMs. However, we found out that such knowledge can be hardly extracted with a carefully designed method tuned on a task specific validation set. According to such finding, we proposed a method (RelBERT) for distilling relational knowledge via LM fine-tuning. This method successfully retrieves flexible relation embeddings that achieve State-of-The-Art (SoTA) in various analogy benchmarks. Moreover, it exhibits a high generalization ability to be able to handle relation types that are not included in the training data. Finally, we propose a new task of modelling graded relation in named entities, which reveals some limitations of recent SoTA LMs as well as RelBERT, suggesting future research direction to model relational knowledge in the current LM era, especially when it comes to named entities.

Item Type:	Thesis (PhD)
Date Type:	Completion
Status:	Unpublished
Schools:	Computer Science & Informatics
Date of First Compliant Deposit:	18 April 2024
Last Modified:	18 Apr 2024 14:57
URI:	https://orca.cardiff.ac.uk/id/eprint/168141

Actions (repository staff only)

Edit Item