用 SCALPEL 剖析 Ullman 变体：为什么 LLM 在对错误信念任务的微小改动上会失败？
Dissecting the Ullman Variations with a SCALPEL: Why do LLMs fail at Trivial Alterations to the False Belief Task?
AI总结：
本文引入 SCALPEL 方法逐步修改刺激以探究 LLM 在心智理论任务微小改动上失败的原因，发现其缺乏常识推理能力，虽超越模式匹配但仍未达到鲁棒的类人心智理论水平。
摘要：
最近的实证结果引发了关于大型语言模型（LLMs）是否具备心智理论（ToM）能力的争论。尽管一些人发现 LLMs 在 ToM 评估（如错误信念任务）上取得了成功，但另一些人表明，它们的性能对刺激的微小改动并不鲁棒。在本文中，我们引入了 SCALPEL——一种逐步修改刺激以测试关于 LLMs 为何失败的不同具体假设的技术——并将该方法应用于意外内容任务的“透明访问”修改版。我们的结果表明，LLMs 通常表现不佳，因为它们无法做出必要的常识推理，例如看到透明容器意味着识别其内容。我们得出结论，尽管现代 LLMs 超越了单纯的模式匹配，但它们仍然缺乏鲁棒的类人 ToM。我们认为，SCALPEL 可以帮助认知科学家更细致地检验 LLMs 的能力，并为完成用于评估人类认知的任务的替代机制提供见解。
英文摘要：
Recent empirical results have sparked a debate about whether or not Large Language Models (LLMs) are capable of Theory of Mind (ToM). While some have found LLMs to be successful on ToM evaluations such as the False Belief task, others have shown that their performance is not robust against trivial alterations to stimuli. In this paper, we introduce SCALPEL -- a technique to incrementally modify stimuli to test different specific hypotheses about why LLMs fail -- and apply this method to the "transparent-access" modification of the unexpected contents task. Our results suggest that LLMs often do poorly because they fail to make essential common-sense inferences, such as that seeing a transparent container implies recognizing its contents. We conclude that while modern LLMs go beyond mere pattern matching, they still fall short of robust human-like ToM. We argue that SCALPEL can help cognitive scientists examine LLMs' capabilities in finer detail and provide insight into alternative mechanisms by which tasks that are used to assess human cognition might be completed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
