arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2107.09948cs.CLstat.AP
收藏

词序演化的统计模型

A Statistical Model of Word Rank Evolution

University of California Merced(加利福尼亚大学默塞德分校);University of California Los Angeles(加利福尼亚大学洛杉矶分校)

Alex John Quijano, Rick Dale, Suzanne Sindi

摘要:

大量语言数据集的可用性使数据驱动的方法能够研究语言变化。Google图书语料库的单字频率数据集被用来调查八种语言的单词排名动态。我们观察了从1900年到2008年的一元图的排名变化,并将其与我们为分析开发的赖特-费舍尔启发模型进行了比较。该模型模拟了一个中性的进化过程,限制没有消失和添加的单词。这项工作解释了该模型的数学框架-写为具有多项式转移概率的马尔可夫链-以显示单词的频率如何随时间变化。从我们对数据和模型的观察来看,单词等级稳定性表现出两种类型的特征:(1)等级的增加/减少是单调的,或者(2)等级保持不变。根据我们的模型,高排名的单词往往更稳定,而低排名的单词往往更不稳定。有些词的等级变化有两种方式:(a)随着时间的推移,等级的小幅度上升/下降变化的积累;(b)等级上升/下降的冲击。我们所研究的所有语言中的大多数单词都是等级稳定的,但并不像中性模型预测的那样稳定。停用词和Swadesh词被观察到在八种语言中排名稳定,表明在既定语言中的语言一致性。这些特征表明,所有语言中的单字频率都以一种与纯粹中性进化过程不一致的方式发生了变化。

英文摘要:

The availability of large linguistic data sets enables data-driven approaches to study linguistic change. The Google Books corpus unigram frequency data set is used to investigate the word rank dynamics in eight languages. We observed the rank changes of the unigrams from 1900 to 2008 and compared it to a Wright-Fisher inspired model that we developed for our analysis. The model simulates a neutral evolutionary process with the restriction of having no disappearing and added words. This work explains the mathematical framework of the model - written as a Markov Chain with multinomial transition probabilities - to show how frequencies of words change in time. From our observations in the data and our model, word rank stability shows two types of characteristics: (1) the increase/decrease in ranks are monotonic, or (2) the rank stays the same. Based on our model, high-ranked words tend to be more stable while low-ranked words tend to be more volatile. Some words change in ranks in two ways: (a) by an accumulation of small increasing/decreasing rank changes in time and (b) by shocks of increase/decrease in ranks. Most words in all of the languages we have looked at are rank stable, but not as stable as a neutral model would predict. The stopwords and Swadesh words are observed to be rank stable across eight languages indicating linguistic conformity in established languages. These signatures suggest unigram frequencies in all languages have changed in a manner inconsistent with a purely neutral evolutionary process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑