心智理论中语境理解的本质：一项基于故事角色的问答研究
The Essence of Contextual Understanding in Theory of Mind: A Study on Question Answering with Story Characters
AI总结：
本文提出 CharToM 基准，利用经典小说角色构建 1,035 个心智理论问题，证明个人背景语境对 ToM 推理至关重要，并揭示当前大语言模型在该任务上仍明显落后于人类。
摘要：
心智理论（Theory-of-Mind, ToM）是一种基础心理能力，使人类能够理解并解释他人的心理状态。人类通过整合来自广泛语境信息的因果线索和间接线索来推断他人的想法，这些信息通常来源于过去的互动。换言之，人类的心智理论在很大程度上依赖于对他人背景和人生经历的理解。遗憾的是，现有用于评估机器心智理论能力的基准大多忽视了这一方面，因为它们使用的是缺乏全局语境、尤其是缺乏角色个人背景的短篇叙述。本文验证了对个人背景进行全面语境理解在心智理论中的重要性，并评估了大语言模型（LLMs）在此类复杂场景中的表现。为此，我们引入了 CharToM 基准，该基准包含 1,035 个基于经典小说角色的心智理论问题。我们的人类研究揭示出显著的性能差异：同一组受过教育的参与者在读过小说时的表现远优于未读过小说时的表现。与此同时，我们对最先进的大语言模型（包括最新的 o1 和 DeepSeek-R1 模型）进行的实验表明，尽管这些模型在预训练阶段已经见过这些故事，其表现仍明显逊于人类。这凸显了当前大语言模型在捕捉心智理论推理所需的细微语境信息方面存在局限性。
英文摘要：
Theory-of-Mind (ToM) is a fundamental psychological capability that allows humans to understand and interpret the mental states of others. Humans infer others' thoughts by integrating causal cues and indirect clues from broad contextual information, often derived from past interactions. In other words, human ToM heavily relies on the understanding about the backgrounds and life stories of others. Unfortunately, this aspect is largely overlooked in existing benchmarks for evaluating machines' ToM capabilities, due to their usage of short narratives without global context, especially personal background of characters. In this paper, we verify the importance of comprehensive contextual understanding about personal backgrounds in ToM and assess the performance of LLMs in such complex scenarios. To achieve this, we introduce CharToM benchmark, comprising 1,035 ToM questions based on characters from classic novels. Our human study reveals a significant disparity in performance: the same group of educated participants performs dramatically better when they have read the novels compared to when they have not. In parallel, our experiments on state-of-the-art LLMs, including the very recent o1 and DeepSeek-R1 models, show that LLMs still perform notably worse than humans, despite that they have seen these stories during pre-training. This highlights the limitations of current LLMs in capturing the nuanced contextual information required for ToM reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
