arXiv 2210.073622023-05-09cs.CL

人口统计因素能否改善文本分类？在Transformer时代重新审视人口统计适配

Can Demographic Factors Improve Text Classification? Revisiting Demographic Adaptation in the Age of Transformers

University of Mannheim（曼海姆大学）；NEC Laboratories Europe GmbH（NEC欧洲实验室有限公司）；University of Hamburg（汉堡大学）；Bocconi University（博科尼大学）；University of Würzburg（维尔茨堡大学）

Chia-Chien Hung, Anne Lauscher, Dirk Hovy, Simone Paolo Ponzetto, Goran Glavaš

备注：: Findings of EACL 2023. arXiv admin note: text overlap with arXiv:2208.01029

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究考察人口统计适配在预训练Transformer语言模型中的有效性，发现表面上的性能提升源于混杂因素而非人口统计知识，表明该问题在现代NLP中仍未解决。

摘要：

人口统计因素（如性别或年龄）塑造了我们的语言。以往研究表明，在传统NLP模型中纳入人口统计因素可以持续提升各种NLP任务的性能。在本研究中，我们探究这些先前发现是否仍然适用于基于预训练Transformer的最先进语言模型（PLMs）。我们采用了三种已被证明有效的、将外部知识（如领域特定或地理知识）融入预训练Transformer的常用专门化方法。我们针对性别和年龄这两个人口统计维度调整语言表示，使用连续语言建模和动态多任务学习进行适配，其中我们将语言建模目标与人口统计类别的预测相结合。当使用多语言PLM时，我们的结果显示在四种语言（英语、德语、法语和丹麦语）的任务性能上取得了显著提升，这与先前工作的结果一致。然而，在控制混杂因素——主要是基于Transformer的PLM的领域和语言熟练度——之后，下游性能的提升实际上并非源于我们的人口统计适配所带来的人口统计知识。我们的结果表明，PLM的人口统计专门化虽然有望产生积极的社会影响，但对于（现代）NLP来说仍然是一个未解决的问题。

英文摘要：

Demographic factors (e.g., gender or age) shape our language. Previous work showed that incorporating demographic factors can consistently improve performance for various NLP tasks with traditional NLP models. In this work, we investigate whether these previous findings still hold with state-of-the-art pretrained Transformer-based language models (PLMs). We use three common specialization methods proven effective for incorporating external knowledge into pretrained Transformers (e.g., domain-specific or geographic knowledge). We adapt the language representations for the demographic dimensions of gender and age, using continuous language modeling and dynamic multi-task learning for adaptation, where we couple language modeling objectives with the prediction of demographic classes. Our results, when employing a multilingual PLM, show substantial gains in task performance across four languages (English, German, French, and Danish), which is consistent with the results of previous work. However, controlling for confounding factors - primarily domain and language proficiency of Transformer-based PLMs - shows that downstream performance gains from our demographic adaptation do not actually stem from demographic knowledge. Our results indicate that demographic specialization of PLMs, while holding promise for positive societal impact, still represents an unsolved problem for (modern) NLP.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。