不同字长频率:分布和符号熵
Distinct word length frequencies: distributions and symbol entropies
摘要:
我们将使用两种方法来分析一种语言的词汇中不同单词的频率计数在长度上的分布。第一,将着眼于几种语言的经验分布,并得出一个合理解释不同单词数量作为长度函数的分布。我们将能够根据字母和空格的边际概率导出频率计数、平均字长和字长的方差。第二部分是基于信息论,证明了条件熵也可以用来估计语言中给定长度的不同词的频率。此外,它将显示如何这些技术也可以应用于估计高阶熵使用词汇字长。
英文摘要:
The distribution of frequency counts of distinct words by length in a language's vocabulary will be analyzed using two methods. The first, will look at the empirical distributions of several languages and derive a distribution that reasonably explains the number of distinct words as a function of length. We will be able to derive the frequency count, mean word length, and variance of word length based on the marginal probability of letters and spaces. The second, based on information theory, will demonstrate that the conditional entropies can also be used to estimate the frequency of distinct words of a given length in a language. In addition, it will be shown how these techniques can also be applied to estimate higher order entropies using vocabulary word length.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。