arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.17786cs.CLcs.LG
收藏

注意力并非你始终所需:面向领域特定文本的高效分类

Attention is Not Always What You Need: Towards Efficient Classification of Domain-Specific Text

Western University(西安大略大学);IBM Canada(IBM加拿大公司)

Yasmen Wahba, Nazim Madhavji, John Steinbacher

AI总结:

针对领域特定文本分类任务,本研究对比前沿预训练语言模型与线性SVM+TFIDF模型在三个数据集上的表现,发现线性模型性能相当且更高效、可解释,为该类任务提供了更优替代选择。

摘要:

对于包含按层级组织的数百个类别的大规模IT语料库而言,准确分类层级中较高级别类别的任务至关重要,可避免错误传播至较低级别。在商业领域,相较于成本高昂的黑盒模型,人们更倾向于选用高效且可解释的机器学习(ML)模型,尤其是在性能提升幅度较小时。自然语言处理(NLP)领域当前的一个趋势是,将大型预训练语言模型(PLMs)或称自注意力模型(例如BERT)应用于几乎所有类型的NLP任务(如问答、情感分析、文本分类)。尽管PLMs应用广泛且在各类NLP任务中表现出色,但目前仍缺乏清晰且充分的依据说明为何要将这些模型用于领域特定文本分类(TC)任务——因为领域特定文本中存在的专业词汇(即行话)具有单义性,这使得上下文化嵌入(如PLMs)的作用失效。本文中,我们将文献中报道的部分前沿(SOTA)模型的准确率,与线性SVM分类器结合TFIDF向量化模型在三个文本分类数据集上的表现进行了对比。结果显示,LinearSVM的性能与之相当。本研究结果表明,对于领域特定文本分类任务,线性模型可提供性能相当、成本低廉、可复现且可解释的替代方案,以替代基于注意力的模型。

英文摘要:

For large-scale IT corpora with hundreds of classes organized in a hierarchy, the task of accurate classification of classes at the higher level in the hierarchies is crucial to avoid errors propagating to the lower levels. In the business world, an efficient and explainable ML model is preferred over an expensive black-box model, especially if the performance increase is marginal. A current trend in the Natural Language Processing (NLP) community is towards employing huge pre-trained language models (PLMs) or what is known as self-attention models (e.g., BERT) for almost any kind of NLP task (e.g., question-answering, sentiment analysis, text classification). Despite the widespread use of PLMs and the impressive performance in a broad range of NLP tasks, there is a lack of a clear and well-justified need to as why these models are being employed for domain-specific text classification (TC) tasks, given the monosemic nature of specialized words (i.e., jargon) found in domain-specific text which renders the purpose of contextualized embeddings (e.g., PLMs) futile. In this paper, we compare the accuracies of some state-of-the-art (SOTA) models reported in the literature against a Linear SVM classifier and TFIDF vectorization model on three TC datasets. Results show a comparable performance for the LinearSVM. The findings of this study show that for domain-specific TC tasks, a linear model can provide a comparable, cheap, reproducible, and interpretable alternative to attention-based models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑