“他们没有教养”：揭示 LLM 生成对话中的隐蔽伤害与社会威胁
"They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversations
AI总结：
本文提出基于社会科学的七项 CHAST 指标，并用与人类判断一致的评估模型检测招聘场景下 LLM 对话中的隐蔽文化伤害，发现多数模型存在难检测的偏见，且对非西方概念更极端。
摘要：
大型语言模型（LLM）已成为现代社会不可或缺的一部分，为个人助理等面向用户的应用以及招聘工具等企业应用提供支持。尽管其实用性很强，但研究表明，LLM 会延续系统性偏见。然而，以往关于 LLM 伤害的研究主要关注种族和性别等西方概念，往往忽视世界其他地区的文化概念。此外，这些研究通常将“伤害”作为单一维度进行考察，忽视了伤害显现的各种微妙形式。为弥补这一空白，我们提出了 Covert Harms and Social Threats（CHAST，隐蔽伤害与社会威胁），这是一组基于社会科学文献的七项指标。我们利用与人类评估相一致的评估模型，考察 LLM 生成对话中是否存在隐蔽伤害，尤其是在招聘场景中。实验表明，本研究纳入的 8 个 LLM 中有 7 个生成了充斥 CHAST 的对话，其特征是以看似中立的语言表达恶意观点，现有方法不太可能检测到这些内容。值得注意的是，与种族等西方概念相比，这些 LLM 在处理种姓等非西方概念时表现出更极端的观点和看法。
英文摘要：
Large language models (LLMs) have emerged as an integral part of modern societies, powering user-facing applications such as personal assistants and enterprise applications like recruitment tools. Despite their utility, research indicates that LLMs perpetuate systemic biases. Yet, prior works on LLM harms predominantly focus on Western concepts like race and gender, often overlooking cultural concepts from other parts of the world. Additionally, these studies typically investigate "harm" as a singular dimension, ignoring the various and subtle forms in which harms manifest. To address this gap, we introduce the Covert Harms and Social Threats (CHAST), a set of seven metrics grounded in social science literature. We utilize evaluation models aligned with human assessments to examine the presence of covert harms in LLM-generated conversations, particularly in the context of recruitment. Our experiments reveal that seven out of the eight LLMs included in this study generated conversations riddled with CHAST, characterized by malign views expressed in seemingly neutral language unlikely to be detected by existing methods. Notably, these LLMs manifested more extreme views and opinions when dealing with non-Western concepts like caste, compared to Western ones such as race.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
