Journal of Data Analysis and Information Processing

Volume 7, Issue 2 (May 2019)

ISSN Print: 2327-7211   ISSN Online: 2327-7203

Google-based Impact Factor: 1.33  Citations  

Measuring Dynamic Correlations of Words in Written Texts with an Autocorrelation Function

HTML  XML Download Download as PDF (Size: 3880KB)  PP. 46-73  
DOI: 10.4236/jdaip.2019.72004    317 Downloads   679 Views  Citations

ABSTRACT

In this study, we regard written texts as time series data and try to investigate dynamic correlations of word occurrences by utilizing an autocorrelation function (ACF). After defining appropriate formula for the ACF that is suitable for expressing the dynamic correlations of words, we use the formula to calculate ACFs for frequent words in 12 books. The ACFs obtained can be classified into two groups: One group of ACFs shows dynamic correlations, with these ACFs well described by a modified Kohlrausch-Williams-Watts (KWW) function; the other group of ACFs shows no correlations, with these ACFs fitted by a simple stepdown function. A word having the former ACF is called a Type-I word and a word with the latter ACF is called a Type-II word. It is also shown that the ACFs of Type-II words can be derived theoretically by assuming that the stochastic process governing word occurrence is a homogeneous Poisson point process. Based on the fitting of the ACFs by KWW and stepdown functions, we propose a measure of word importance which expresses the extent to which a word is important in a particular text. The validity of the measure is confirmed by using the Kleinburg’s burst detection algorithm.

Share and Cite:

Ogura, H. , Amano, H. and Kondo, M. (2019) Measuring Dynamic Correlations of Words in Written Texts with an Autocorrelation Function. Journal of Data Analysis and Information Processing, 7, 46-73. doi: 10.4236/jdaip.2019.72004.

Copyright © 2020 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.