arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1812.01207cs.CL
收藏

基于预训练语言模型的实用文本分类

Practical Text Classification With Large Pre-Trained Language Models

University of California, Berkeley(加州大学伯克利分校);NVIDIA(英伟达)

Neel Kant, Raul Puri, Nikolai Yakovenko, Bryan Catanzaro

摘要:

多情感情感分类是一个自然语言处理(NLP)问题,具有真实世界数据的有价值用例。我们证明了大规模无监督语言建模与微调相结合,为困难数据集上的这一任务提供了一个实用的解决方案,包括那些具有标签类不平衡和特定领域上下文的数据集。通过在40 GB的文本(亚马逊评论)(McAuley et al. 2015)上训练基于注意力的Transformer网络(Vaswani et al. 2017)并对训练集进行微调,我们的模型在SemEval任务1:E-c多维情感分类问题(Mohammad et al. 2018)上获得了0.69 F1分数,基于Plutchik轮情感(Plutchik 1979)。这些结果与最先进的模型相比具有竞争力,包括在困难(情绪)类别(如恐惧(0.73),厌恶(0.77)和愤怒(0.78))上的强大F1分数,以及在罕见类别(如预期(0.42)和惊喜(0.37))上的竞争结果。此外,我们展示了我们的应用程序在现实世界中的文本分类任务。我们创建了一个严格收集的关于几个主题的真实推文的文本数据集,并表明我们的微调模型在此数据集上的情感和多维情感分类方面优于通用商业API。我们还进行了各种额外的研究,调查深度学习架构,数据集和算法的属性,以实现实用的多维情感分类。总的来说,我们发现无监督语言建模和微调是一个简单的框架,可以在现实世界的情感分类中获得高质量的结果。

英文摘要:

Multi-emotion sentiment classification is a natural language processing (NLP) problem with valuable use cases on real-world data. We demonstrate that large-scale unsupervised language modeling combined with finetuning offers a practical solution to this task on difficult datasets, including those with label class imbalance and domain-specific context. By training an attention-based Transformer network (Vaswani et al. 2017) on 40GB of text (Amazon reviews) (McAuley et al. 2015) and fine-tuning on the training set, our model achieves a 0.69 F1 score on the SemEval Task 1:E-c multi-dimensional emotion classification problem (Mohammad et al. 2018), based on the Plutchik wheel of emotions (Plutchik 1979). These results are competitive with state of the art models, including strong F1 scores on difficult (emotion) categories such as Fear (0.73), Disgust (0.77) and Anger (0.78), as well as competitive results on rare categories such as Anticipation (0.42) and Surprise (0.37). Furthermore, we demonstrate our application on a real world text classification task. We create a narrowly collected text dataset of real tweets on several topics, and show that our finetuned model outperforms general purpose commercially available APIs for sentiment and multidimensional emotion classification on this dataset by a significant margin. We also perform a variety of additional studies, investigating properties of deep learning architectures, datasets and algorithms for achieving practical multidimensional sentiment classification. Overall, we find that unsupervised language modeling and finetuning is a simple framework for achieving high quality results on real-world sentiment classification.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑