arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1902.10339cs.CLcs.LG
收藏

文本分类需要多大的词汇量?词汇选择的变分方法

How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection

University of California, Santa Barbara(加州大学圣巴巴拉分校);Ohio State University(俄亥俄州立大学);Samsung Research(三星研究院)

Wenhu Chen, Yu Su, Yilin Shen, Zhiyu Chen, Xifeng Yan, William Wang

摘要:

随着深度学习的快速发展,深度神经网络已被广泛应用于许多现实生活中的自然语言应用中。在深度神经网络下,需要预定义的词汇表来对文本输入进行矢量化。预定义词汇的规范选择方法是基于词频的,其中选择阈值来截断长尾分布。然而,我们观察到这种简单的方法很容易导致词汇量不足或词汇量过大的问题。因此,我们有兴趣了解最终任务分类精度与词汇量的关系,以及达到特定性能所需的最小词汇量是多少。在变分丢弃算法的基础上,提出了一种更复杂的变分词汇丢弃算法(VVD)来进行词汇选择,它可以智能地选择词汇子集以达到所需的性能。为了评估新提出的词汇选择问题的不同算法,我们提出了两个新的指标:准确度-词汇曲线下的面积和X\%准确度下降下的词汇大小。通过对各种NLP分类任务的广泛实验,我们的变分框架在这些指标上显着优于基于频率和其他选择基线。

英文摘要:

With the rapid development in deep learning, deep neural networks have been widely adopted in many real-life natural language applications. Under deep neural networks, a pre-defined vocabulary is required to vectorize text inputs. The canonical approach to select pre-defined vocabulary is based on the word frequency, where a threshold is selected to cut off the long tail distribution. However, we observed that such simple approach could easily lead to under-sized vocabulary or over-sized vocabulary issues. Therefore, we are interested in understanding how the end-task classification accuracy is related to the vocabulary size and what is the minimum required vocabulary size to achieve a specific performance. In this paper, we provide a more sophisticated variational vocabulary dropout (VVD) based on variational dropout to perform vocabulary selection, which can intelligently select the subset of the vocabulary to achieve the required performance. To evaluate different algorithms on the newly proposed vocabulary selection problem, we propose two new metrics: Area Under Accuracy-Vocab Curve and Vocab Size under X\% Accuracy Drop. Through extensive experiments on various NLP classification tasks, our variational framework is shown to significantly outperform the frequency-based and other selection baselines on these metrics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑