arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1305.0638cs.LGcs.IRstat.ML
收藏

基于词频和T检验的文本分类特征选择

Feature Selection Based on Term Frequency and T-Test for Text Categorization

Beihang University(北京航空航天大学)

Deqing Wang, Hui Zhang, Rui Liu, Weifeng Lv

摘要:

在特征选择方面已经做了大量的工作。现有的方法都是基于文档频率的,如卡方统计、信息增益等,但这些方法存在两个不足:一是对低频词不可靠;二是只统计某个词是否出现在文档中,而忽略了词的频率。事实上,在特定的范畴内的高频术语通常被认为是歧视。本文研究了如何构造基于词频的特征选择函数,并提出了一种基于$t$ -test的特征选择方法,该方法用于衡量术语在特定类别和整个语料库中的分布差异。使用三个分类器在两个文本语料库上进行的广泛的比较实验表明,我们的新方法与最先进的特征选择方法(即,$χ^2$和IG)在宏观$F_1$和微观$F_1$方面。

英文摘要:

Much work has been done on feature selection. Existing methods are based on document frequency, such as Chi-Square Statistic, Information Gain etc. However, these methods have two shortcomings: one is that they are not reliable for low-frequency terms, and the other is that they only count whether one term occurs in a document and ignore the term frequency. Actually, high-frequency terms within a specific category are often regards as discriminators. This paper focuses on how to construct the feature selection function based on term frequency, and proposes a new approach based on $t$-test, which is used to measure the diversity of the distributions of a term between the specific category and the entire corpus. Extensive comparative experiments on two text corpora using three classifiers show that our new approach is comparable to or or slightly better than the state-of-the-art feature selection methods (i.e., $χ^2$, and IG) in terms of macro-$F_1$ and micro-$F_1$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑