Frequency consolidation among word N-grams: a practical procedure

Buerki, Andreas ORCID: https://orcid.org/0000-0003-2151-3246 2017. Frequency consolidation among word N-grams: a practical procedure. Mitkov, Ruslan, ed. Computational and Corpus-Based Phraseology, Lecture notes in Computer Science, vol. 10596. Cham: Springer, pp. 432-446. (10.1007/978-3-319-69805-2)

Preview

PDF - Published Version Download (365kB) | Preview

Abstract

This paper considers the issue of frequency consolidation in lists of different length word n-grams (i.e. recurrent word sequences) extracted from the same underlying corpus. A simple algorithm – enhanced by a preparatory stage – is proposed which allows the con- solidation of frequencies among lists of different length n-grams, from 2-grams to 6-grams and beyond. The consolidation adjusts the frequency count of each n-gram to the number of its occurrences minus its occur- rences as part of longer n-grams. Among other uses, such a procedure aids linguistic analysis and allows the non-inflationary counting of word tokens that are part of frequent n-grams of various lengths, which in turn allows an assessment of the proportion of running text made up of recurring chunks. The proposed procedure delivers frequency consolida- tion and substring reduction among word n-grams and is independent of any particular method of n-gram extraction and filtering, making it applicable also in situations where full access to underlying corpora is unavailable.

Item Type:	Book Section
Date Type:	Publication
Status:	Published
Schools:	English, Communication and Philosophy
Subjects:	P Language and Literature > P Philology. Linguistics
Publisher:	Springer
ISBN:	9783319698045
Date of First Compliant Deposit:	9 November 2017
Last Modified:	03 Nov 2022 09:54
URI:	https://orca.cardiff.ac.uk/id/eprint/106389

Actions (repository staff only)

Edit Item