arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.04445cs.CL
收藏

Classist Tools: 社会阶层与NLP性能的相关性

Classist Tools: Social Class Correlates with Performance in NLP

MilaNLP(米拉NLP);Bocconi University(博科尼大学);Instituto de Telecomunicações(电信研究所);University of Artificial Intelligence(人工智能大学)

Amanda Cercas Curry, Giuseppe Attanasio, Zeerak Talat, Dirk Hovy

AI总结:

该研究通过标注含9.5万条电影台词的多维度语料库,实证发现NLP系统在三项任务中对低社会经济地位等边缘群体存在性能差异,提出语言技术需纳入社会阶层因素以保障公平。

摘要:

自William Labov关于语言社会分层的奠基性研究(Labov, 1964)以来,语言学界一直集中探索社会人口特征与语言产出、感知之间的关联。但尽管有充分证据表明语言中存在社会人口特征,这些特征却很少被用于自然语言处理(NLP)领域。年龄和性别在相关研究中尚有一定体现,但Labov最初关注的社会经济地位却明显缺失。而这一因素至关重要。我们通过实证研究表明,NLP对社会经济地位较低的群体存在不利影响。我们标注了一个包含9.5万条电影台词的语料库,标注维度包括社会阶层、族裔和地理语言变体,并在三项任务上衡量NLP系统的性能:语言建模、自动语音识别和语法错误纠正。我们发现,可归因于社会经济地位以及族裔、地理差异的性能差异十分显著。随着NLP技术日益普及和日常化,这类技术必须适配所有语言变体,避免让本就处于边缘地位的群体处于不利境地。我们主张未来的语言技术应纳入社会经济阶层因素。

英文摘要:

Since the foundational work of William Labov on the social stratification of language (Labov, 1964), linguistics has made concentrated efforts to explore the links between sociodemographic characteristics and language production and perception. But while there is strong evidence for socio-demographic characteristics in language, they are infrequently used in Natural Language Processing (NLP). Age and gender are somewhat well represented, but Labov's original target, socioeconomic status, is noticeably absent. And yet it matters. We show empirically that NLP disadvantages less-privileged socioeconomic groups. We annotate a corpus of 95K utterances from movies with social class, ethnicity and geographical language variety and measure the performance of NLP systems on three tasks: language modelling, automatic speech recognition, and grammar error correction. We find significant performance disparities that can be attributed to socioeconomic status as well as ethnicity and geographical differences. With NLP technologies becoming ever more ubiquitous and quotidian, they must accommodate all language varieties to avoid disadvantaging already marginalised groups. We argue for the inclusion of socioeconomic class in future language technologies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑