arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2004.06474cs.CLcs.DSphysics.soc-ph
收藏

有意义文本的两个半部分在统计上是不同的

Two halves of a meaningful text are statistically different

Central China Normal University(华中师范大学);Yerevan Physics Institute(埃里温物理研究所)

Weibing Deng, R. Xie, S. Deng, Armen E. Allahverdyan

AI总结:

本文通过比较文本前后两半的词汇与分布特征,发现有意义文本存在稳定的时间不对称性,并将其与叙事驱动的语义组织相联系。

摘要:

哪些统计特征能将一段有意义的文本(可能以未知系统书写)与无意义的符号集合区分开来?本文通过比较文本前半部分与后半部分的特征来回答这一问题。这种比较可以揭示隐藏效应,因为两个半部分在许多参数(风格、体裁等)上具有相同的值。我们发现,前半部分比后半部分拥有更多不同的词和更多的稀有词。此外,就频率与逆空间周期之间的差异而言,前半部分中的词在整个文本中的分布更不均匀。在我们研究的数百篇相对较短的文本中,绝大多数都存在这些差异。统计显著性通过 Wilcoxon 检验得到确认。在对词进行随机排列、破坏文本的线性结构之后,这些差异便消失了。这些差异揭示了有意义文本中的时间不对称性;这一点通过以下事实得到证实:文本以其自然方式(即沿叙事方向)的可压缩性远高于词序反转形式。我们推测,这些结果将文本的语义组织(由其叙事流定义)与其统计特征联系起来。

英文摘要:

Which statistical features distinguish a meaningful text (possibly written in an unknown system) from a meaningless set of symbols? Here we answer this question by comparing features of the first half of a text to its second half. This comparison can uncover hidden effects, because the halves have the same values of many parameters (style, genre {\it etc}). We found that the first half has more different words and more rare words than the second half. Also, words in the first half are distributed less homogeneously over the text in the sense of of the difference between the frequency and the inverse spatial period. These differences hold for the significant majority of several hundred relatively short texts we studied. The statistical significance is confirmed via the Wilcoxon test. Differences disappear after random permutation of words that destroys the linear structure of the text. The differences reveal a temporal asymmetry in meaningful texts, which is confirmed by showing that texts are much better compressible in their natural way (i.e. along the narrative) than in the word-inverted form. We conjecture that these results connect the semantic organization of a text (defined by the flow of its narrative) to its statistical features.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑