MoCa：测量人类与语言模型在因果与道德判断任务上的对齐程度
MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks
AI总结：
本文构建了基于24篇认知科学论文的MoCa数据集，用于测试大语言模型在因果与道德判断上与人类的对齐程度，发现总体对齐虽随模型更新而提高，但模型对各影响因素的权衡与人类存在显著差异。
摘要：
人类对物理世界和社会世界的常识理解是围绕直觉理论组织起来的。这些理论支持人们做出因果判断和道德判断。当坏事发生时，我们会自然地追问：谁做了什么，为什么？认知科学领域的大量文献已经研究了人们的因果直觉和道德直觉。这些研究揭示了一系列系统性地影响人们判断的因素，例如对规范的违反，以及伤害是可避免的还是不可避免的。我们收集了一个来自24篇认知科学论文的故事数据集，并开发了一个系统来为每个故事标注这些论文所研究的因素。利用该数据集，我们测试了大语言模型（LLMs）在基于文本的场景中做出的因果判断和道德判断是否与人类参与者的判断相对齐。在总体层面上，随着较新的大语言模型的出现，对齐程度有所提高。然而，通过统计分析，我们发现大语言模型对不同因素的权衡方式与人类参与者存在很大差异。这些结果表明，经过精心策划的挑战性数据集与认知科学的洞见相结合，能够帮助我们超越仅仅基于总体指标的比较：我们揭示了大语言模型的隐性倾向，并展示了这些倾向在多大程度上与人类直觉相对齐。
英文摘要：
Human commonsense understanding of the physical and social world is organized around intuitive theories. These theories support making causal and moral judgments. When something bad happens, we naturally ask: who did what, and why? A rich literature in cognitive science has studied people's causal and moral intuitions. This work has revealed a number of factors that systematically influence people's judgments, such as the violation of norms and whether the harm is avoidable or inevitable. We collected a dataset of stories from 24 cognitive science papers and developed a system to annotate each story with the factors they investigated. Using this dataset, we test whether large language models (LLMs) make causal and moral judgments about text-based scenarios that align with those of human participants. On the aggregate level, alignment has improved with more recent LLMs. However, using statistical analyses, we find that LLMs weigh the different factors quite differently from human participants. These results show how curated, challenge datasets combined with insights from cognitive science can help us go beyond comparisons based merely on aggregate metrics: we uncover LLMs implicit tendencies and show to what extent these align with human intuitions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
