arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.26089cs.LGcs.AIcs.CL
收藏

对LLM心理自我建模能力的有选择性缺陷的测试:基于行为的理论自我认知测试

Selective Deficits in LLM Mental Self-Modeling in a Behavior-Based Test of Theory of Mind

Christopher Ackerman

AI总结:

本文通过行为测试探讨LLM在理论自我认知任务中的能力,发现较早的LLM无法完成任务,而较新的LLM在他人认知建模上表现接近人类,但自我建模仍存在缺陷,除非使用推理轨迹辅助。

摘要:

能够将自己和他人视为具有知识、意图和信念状态的代理,从而引导其行为的能力——理论自我认知——是人类普遍具备的能力,使我们能够导航并操控社会世界。这种能力依赖于我们形成自己和他人心理模型的能力。其在人类事务中的普遍性意味着LLM在其训练数据中已见过无数实例,因此可能已学会模仿,但它们是否真的学习了可以应用于任意设置的因果模型尚不清楚。因此,我们开发了一种新的实验范式,要求受试者形成自己和他人的心理状态的表示,并战略性地行动,而非仅仅描述。我们测试了自2024年以来发布的各种领先开源和闭源LLM以及人类受试者。我们发现:1)2025年中之前发布的LLM在所有任务中均失败;2)更近期的LLM在建模他人认知状态方面达到人类水平;3)即使是最前沿的LLM在自我建模任务中也失败,除非提供推理轨迹作为辅助。我们进一步展示了认知负荷对他人建模任务的影响,提供了暗示证据,表明LLM可能在单次正向传递中使用类似有限容量的工作记忆来保持这些心理表征。最后,我们探讨了推理模型在自我和他人建模任务中成功机制,并显示它们能够轻易地进行战略性欺骗。

英文摘要:

The ability to represent oneself and others as agents with knowledge, intentions, and belief states that guide their behavior - Theory of Mind - is a human universal that enables us to navigate - and manipulate - the social world. It is supported by our ability to form mental models of ourselves and others. Its ubiquity in human affairs entails that LLMs have seen innumerable examples of it in their training data and therefore may have learned to mimic it, but whether they have actually learned causal models that they can deploy in arbitrary settings is unclear. We therefore develop a novel experimental paradigm that requires that subjects form representations of the mental states of themselves and others and act on them strategically rather than merely describe them. We test a wide range of leading open and closed source LLMs released since 2024, as well as human subjects, on this paradigm. We find that 1) LLMs released before mid-2025 fail at all of our tasks, 2) more recent LLMs achieve human-level performance on modeling the cognitive states of others, and 3) even frontier LLMs fail at our self-modeling task - unless afforded a scratchpad in the form of a reasoning trace. We further demonstrate cognitive load effects on other-modeling tasks, offering suggestive evidence that LLMs are using something akin to limited-capacity working memory to hold these mental representations in mind during a single forward pass. Finally, we explore the mechanisms by which reasoning models succeed at the self- and other-modeling tasks, and show that they readily engage in strategic deception.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑