超几何检验在标准文本分析任务上的表现与 TF-IDF 相当
The hypergeometric test performs comparably to TF-IDF on standard text analysis tasks
AI总结：
本文通过真实文档检索、摘要与分类任务的实证研究，发现 TF-IDF 与超几何检验 P 值负对数高度对应，为从统计显著性角度解释其有效性提供依据。
摘要：
词频—逆文档频率（Term frequency-inverse document frequency，简称 TF-IDF）及其众多变体构成了一类术语加权函数，其成员被广泛应用于文本分析。尽管 TF-IDF 最初是作为一种启发式方法提出的，但研究者已经基于信息论、概率论以及“偏离随机性”范式提出了理论依据。在本工作中，我们呈现了一项实证研究，结果表明：在选定的真实数据文档检索、摘要和分类任务上，TF-IDF 与统计显著性超几何检验几乎相对应。这些发现表明，TF-IDF 与超几何检验 P 值（即超几何分布尾部概率）的负对数之间存在的根本数学联系仍有待阐明。我们将本文的实证分析作为从统计显著性检验视角解释 TF-IDF 长期有效性的第一步。我们希望这些结果能为在文本分析应用中系统评估由显著性检验导出的术语加权函数打开大门。
英文摘要：
Term frequency-inverse document frequency, or TF-IDF for short, and its many variants form a class of term weighting functions the members of which are widely used in text analysis applications. While TF-IDF was originally proposed as a heuristic, theoretical justifications grounded in information theory, probability, and the divergence from randomness paradigm have been advanced. In this work, we present an empirical study showing that TF-IDF corresponds very nearly with the hypergeometric test of statistical significance on selected real-data document retrieval, summarization, and classification tasks. These findings suggest that a fundamental mathematical connection between TF-IDF and the negative logarithm of the hypergeometric test P-value (i.e., a hypergeometric distribution tail probability) remains to be elucidated. We advance the empirical analyses herein as a first step toward explaining the long-standing effectiveness of TF-IDF from a statistical significance testing lens. It is our aspiration that these results will open the door to the systematic evaluation of significance testing derived term weighting functions in text analysis applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
