使用日常道德困境对大型语言模型进行规范性评估
Normative Evaluation of Large Language Models with Everyday Moral Dilemmas
AI总结：
针对现有研究过度简化日常道德困境的问题，该研究基于Reddit的AITA社区超过10,000个道德困境评估了7个大型语言模型，发现其道德判断与人类差异显著且模型间一致性低，凸显了在人工系统中进行伦理评估的必要性。
摘要：
大型语言模型（LLMs）的快速采用，激发了对其编码的道德规范和决策过程的广泛研究。此类研究大多依赖于向 LLMs 提出调查式问题，以评估模型与特定人口群体、道德信仰或政治意识形态的契合程度。虽然这些方法具有参考价值，但它们依赖于相对表面的结构，往往会过度简化日常道德困境背后的复杂性与细微差别。我们认为，沿着更详细的人类互动维度对 LLMs 进行审计至关重要，以便更好地评估它们影响人类信念和行为的程度。为此，我们使用来自 Reddit 上“Am I the Asshole”（AITA）社区的复杂日常道德困境来评估 LLMs，该社区的用户会就日常冲突向其他社区成员寻求道德评判。我们提示七个 LLMs 对超过 10,000 个 AITA 道德困境进行责任归咎并提供解释。随后，我们将 LLMs 的判断与解释与 Reddit 用户以及彼此之间的判断进行比较，旨在揭示其道德推理的模式。结果表明，大型语言模型表现出独特的道德判断模式，与 AITA 子版块上的人类评估存在显著差异。LLMs 表现出中等到高度的自我一致性，但模型间的一致性较低。对模型解释的进一步分析显示，模型在援引各种道德原则时呈现出不同的模式。这些发现凸显了在人工系统中实现一致道德推理的复杂性，以及仔细评估不同模型如何进行伦理判断的必要性。随着 LLMs 继续被用于需要伦理决策的角色，如治疗师和伴侣，仔细评估对于减轻潜在偏见和局限性至关重要。
英文摘要：
The rapid adoption of large language models (LLMs) has spurred extensive research into their encoded moral norms and decision-making processes. Much of this research relies on prompting LLMs with survey-style questions to assess how well models are aligned with certain demographic groups, moral beliefs, or political ideologies. While informative, the adherence of these approaches to relatively superficial constructs tends to oversimplify the complexity and nuance underlying everyday moral dilemmas. We argue that auditing LLMs along more detailed axes of human interaction is of paramount importance to better assess the degree to which they may impact human beliefs and actions. To this end, we evaluate LLMs on complex, everyday moral dilemmas sourced from the "Am I the Asshole" (AITA) community on Reddit, where users seek moral judgments on everyday conflicts from other community members. We prompted seven LLMs to assign blame and provide explanations for over 10,000 AITA moral dilemmas. We then compared the LLMs' judgments and explanations to those of Redditors and to each other, aiming to uncover patterns in their moral reasoning. Our results demonstrate that large language models exhibit distinct patterns of moral judgment, varying substantially from human evaluations on the AITA subreddit. LLMs demonstrate moderate to high self-consistency but low inter-model agreement. Further analysis of model explanations reveals distinct patterns in how models invoke various moral principles. These findings highlight the complexity of implementing consistent moral reasoning in artificial systems and the need for careful evaluation of how different models approach ethical judgment. As LLMs continue to be used in roles requiring ethical decision-making such as therapists and companions, careful evaluation is crucial to mitigate potential biases and limitations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
