arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1606.06996cs.CL
收藏

自然语言的词熵

The word entropy of natural languages

University of Tübingen(蒂宾根大学);University of Cambridge(剑桥大学)

Christian Bentz, Dimitrios Alikaniotis

摘要:

与单词相关的平均不确定性是定量和计算语言学核心的信息论概念。熵已经被确立为这种平均不确定性的度量-也称为平均信息含量。我们在这里使用21种语言的平行文本来建立单词熵收敛到稳定值的标记数量。然后,这些收敛点用于从大规模并行语料库中选择文本,并估计1000多种语言的词熵。我们的研究结果有助于建立定量的语言比较,了解多语言翻译系统的性能,并规范语义相似性措施。

英文摘要:

The average uncertainty associated with words is an information-theoretic concept at the heart of quantitative and computational linguistics. The entropy has been established as a measure of this average uncertainty - also called average information content. We here use parallel texts of 21 languages to establish the number of tokens at which word entropies converge to stable values. These convergence points are then used to select texts from a massively parallel corpus, and to estimate word entropies across more than 1000 languages. Our results help to establish quantitative language comparisons, to understand the performance of multilingual translation systems, and to normalize semantic similarity measures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑