使用随机矩阵理论方法定制停止词的确定
Customized determination of stop words using Random Matrix Theory approach
摘要：
研究了以词为单位计算的词间距离，并与随机矩阵理论（RMT）的分布进行了比较。研究发现，单参数Brody分布可以很好地描述词间距离的分布。使用Brody分布拟合，我们发现一组文本中给定单词之间的距离可以显示混合动态，共存的规则和混沌政权。研究发现，在一定的拟合优度阈值下，Brody分布正确拟合的分布可以被识别为停用词，通常被认为是文本中的无信息部分。通过应用不同的阈值的拟合优度，我们可以提取无信息的话，从分析下的文本到所需的程度。在此基础上，我们制定了一个完全不可知的配方，可用于创建一个定制的一组停止字的文本在任何语言的基础上的话。
英文摘要：
The distances between words calculated in word units are studied and compared with the distributions of the Random Matrix Theory (RMT). It is found that the distribution of distance between the same words can be well described by the single-parameter Brody distribution. Using the Brody distribution fit, we found that the distance between given words in a set of texts can show mixed dynamics, coexisting regular and chaotic regimes. It is found that distributions correctly fitted by the Brody distribution with a certain goodness of the fit threshold can be identifid as stop words, usually considered as the uninformative part of the text. By applying various threshold values for the goodness of fit, we can extract uninformative words from the texts under analysis to the desired extent. On this basis we formulate a fully agnostic recipe that can be used in the creation of a customized set of stop words for texts in any language based on words.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
