论希尔伯格定律及其与吉罗定律的联系
On Hilberg's Law and Its Links with Guiraud's Law
摘要:
Hilberg(1990)假设随机人类文本的有限阶超额熵与文本长度的平方根成正比。假设Hilberg的假设是正确的,我们得到Guiraud定律,它指出文本中的词类型的数量大于文本长度的平方根的比例。我们的推导是基于编码理论中的一些数学猜想和一些实验,这些实验表明,词可以近似地定义为文本的最短上下文无关文法的非终结符。这样的词的操作定义甚至可以应用于没有空格的文本,这不允许曼德尔布罗特对齐普夫定律和吉罗定律的“间歇性沉默”解释。与Mandelbrot的相反,我们的模型假设了人类叙述中的一些概率性长记忆效应,并且可能能够解释Menzerath定律。
英文摘要:
Hilberg (1990) supposed that finite-order excess entropy of a random human text is proportional to the square root of the text length. Assuming that Hilberg's hypothesis is true, we derive Guiraud's law, which states that the number of word types in a text is greater than proportional to the square root of the text length. Our derivation is based on some mathematical conjecture in coding theory and on several experiments suggesting that words can be defined approximately as the nonterminals of the shortest context-free grammar for the text. Such operational definition of words can be applied even to texts deprived of spaces, which do not allow for Mandelbrot's ``intermittent silence'' explanation of Zipf's and Guiraud's laws. In contrast to Mandelbrot's, our model assumes some probabilistic long-memory effects in human narration and might be capable of explaining Menzerath's law.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。