语言模型可能并不理解你：通过故事提示评估理论自我意识
Language Models Might Not Understand You: Evaluating Theory of Mind via Story Prompting
AI总结：
通过故事提示框架评估大语言模型的理论自我意识和世界建模能力，发现模型在世界建模任务中表现优于理论自我意识任务，且对人物推理更准确。
摘要：
我们介绍了StorySim，一个可编程框架，用于合成故事以评估大语言模型的理论自我意识（ToM）和世界建模（WM）能力。与以往可能受预训练数据污染或依赖LLM生成的基准不同，StorySim通过高度可控的Storyboard生成新颖、可组合的故事提示，能够精确操控角色视角和事件。我们利用此框架设计了第一和第二阶理论自我意识任务及世界建模任务，以控制对跟踪和建模心理状态的能力。在一系列LLM上的实验显示，大多数模型在世界建模任务中表现优于理论自我意识任务，且模型在推理主体为人物而非无生命物体时推理更准确。此外，我们的框架使我们发现了启发式行为和对故事早期事件的过度依赖的证据。所有生成数据和评估的代码均免费提供。
英文摘要：
We introduce StorySim, a programmable framework for synthetically generating stories to evaluate the theory of mind (ToM) and world modeling (WM) capabilities of large language models (LLMs). Unlike prior benchmarks that may suffer from contamination in pretraining data, or rely on an LLM for generation, StorySim produces novel, compositional story prompts anchored by a highly controllable Storyboard, enabling precise manipulation of character perspectives and events. Using StorySim, we evaluate LLMs across three kinds of ToM tasks: false belief, goal-directed, and preference attribution tasks. We then use our framework to design first- and second-order ToM tasks alongside WM tasks that control for the ability to track and model mental states. Our experiments across a suite of LLMs show that most models achieve higher accuracy on WM tasks than on ToM tasks, and that some models tend to reason more accurately when the subject of reasoning is a person rather than an inanimate object. Additionally, models struggle with goal-directed reasoning the most, and performance does not always scale with model size. All code for generating data and evaluations is freely available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
