arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.02111cs.CLstat.AP
收藏

基于Transformers的社会科学文本分析神经迁移学习方法

Introduction to Neural Transfer Learning with Transformers for Social Science Text Analysis

Ludwig-Maximilians-Universität München(慕尼黑大学)

Sandra Wankmüller

摘要:

基于transformer的迁移学习模型具有在相对较少的训练数据实例的情况下实现基于文本的监督学习任务的高预测精度的潜力。因此,这些模型很可能有利于社会科学家,寻求尽可能准确的基于文本的措施,但只有有限的资源来注释训练数据。为了使社会科学家能够利用这些潜在的好处,他们的研究,本文解释了这些方法是如何工作的,为什么他们可能是有利的,以及他们的局限性是什么。此外,三个基于transformer的迁移学习模型,BERT(Devlin et al. 2019),RoBERTa(Liu et al. 2019)和Longformer(Beltagy et al. 2020),在三个应用程序上与传统的机器学习算法进行了比较。在所有评估的任务、文本风格和训练数据集大小中,传统模型的表现始终优于使用Transformers的迁移学习,从而证明了这些模型可以为基于文本的社会科学研究带来的好处。

英文摘要:

Transformer-based models for transfer learning have the potential to achieve high prediction accuracies on text-based supervised learning tasks with relatively few training data instances. These models are thus likely to benefit social scientists that seek to have as accurate as possible text-based measures but only have limited resources for annotating training data. To enable social scientists to leverage these potential benefits for their research, this paper explains how these methods work, why they might be advantageous, and what their limitations are. Additionally, three Transformer-based models for transfer learning, BERT (Devlin et al. 2019), RoBERTa (Liu et al. 2019), and the Longformer (Beltagy et al. 2020), are compared to conventional machine learning algorithms on three applications. Across all evaluated tasks, textual styles, and training data set sizes, the conventional models are consistently outperformed by transfer learning with Transformers, thereby demonstrating the benefits these models can bring to text-based social science research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑