arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 0911.2284cs.CL
收藏

英语书面语的经典熵研究

A New Look at the Classical Entropy of Written English

Universidad del Valle(德尔瓦列大学)

Fabio G. Guerrero

摘要:

本文提出了一种用计算机直接处理并根据Shannon理论从第一性原理求出合理长度英文文本样本的熵和冗余的简单方法。作为一个例子,英语语言的熵的结果已经获得了基于书面英语的2030万字符,考虑从一个到五百个字符长度的符号。除了对英语熵的现实价值进行了分析外,还对熵的一些经典概念提出了新的看法。这种方法也可以推广到其他拉丁语言。一些实际应用的影响,如剽窃检测软件,和最少的话,应该使用在社会互联网网络消息,进行了讨论。

英文摘要:

A simple method for finding the entropy and redundancy of a reasonable long sample of English text by direct computer processing and from first principles according to Shannon theory is presented. As an example, results on the entropy of the English language have been obtained based on a total of 20.3 million characters of written English, considering symbols from one to five hundred characters in length. Besides a more realistic value of the entropy of English, a new perspective on some classic entropy-related concepts is presented. This method can also be extended to other Latin languages. Some implications for practical applications such as plagiarism-detection software, and the minimum number of words that should be used in social Internet network messaging, are discussed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑