多样性、密度和同质性：文本集的定量特征指标
Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collections
摘要：
通过定量措施总结数据样本有着悠久的历史，描述性统计就是一个很好的例子。然而，随着自然语言处理方法的蓬勃发展，仍然没有足够的特征度量来描述文本集合中的单词、句子或段落。在这项工作中，我们提出的多样性，密度和同质性，定量测量的分散性，稀疏性和均匀性的文本集合的指标。我们进行了一系列模拟，以验证每个指标都具有所需的属性，并与人类的直觉产生共鸣。在真实数据集上的实验表明，所提出的特征度量与著名模型BERT的文本分类性能高度相关，这可能会启发未来的应用。
英文摘要：
Summarizing data samples by quantitative measures has a long history, with descriptive statistics being a case in point. However, as natural language processing methods flourish, there are still insufficient characteristic metrics to describe a collection of texts in terms of the words, sentences, or paragraphs they comprise. In this work, we propose metrics of diversity, density, and homogeneity that quantitatively measure the dispersion, sparsity, and uniformity of a text collection. We conduct a series of simulations to verify that each metric holds desired properties and resonates with human intuitions. Experiments on real-world datasets demonstrate that the proposed characteristic metrics are highly correlated with text classification performance of a renowned model, BERT, which could inspire future applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
