大语言模型（LLMs）近期展现出心理理论（ToM）能力的潜力与涌现，甚至在某些ToM任务中超越人类。为评估并扩展LLMs的ToM推理能力边界，我们提出了一种新颖的概念、分类法和框架，即基于零、有限与无限信念历史的ToM推理，并开发了一款多轮文本游戏，称为“Pick the Right Stuff”，作为基准。我们使用该游戏评估了六个LLMs，发现它们在零信念历史上的表现始终优于在有限信念历史上的表现。此外，我们发现两个参数规模较小的模型超越了所有参数规模较大的被评估模型。我们期望这项工作为未来ToM基准的开发铺平道路，并促进和推动更复杂的人工智能智能体或系统的发展，这些智能体或系统需要具备更复杂的ToM推理能力。
英文摘要：
Large Language Models (LLMs) have recently shown a promise and emergence of Theory of Mind (ToM) ability and even outperform humans in certain ToM tasks. To evaluate and extend the boundaries of the ToM reasoning ability of LLMs, we propose a novel concept, taxonomy, and framework, the ToM reasoning with Zero, Finite, and Infinite Belief History and develop a multi-round text-based game, called $\textit{Pick the Right Stuff}$, as a benchmark. We have evaluated six LLMs with this game and found their performance on Zero Belief History is consistently better than on Finite Belief History. In addition, we have found two of the models with small parameter sizes outperform all the evaluated models with large parameter sizes. We expect this work to pave the way for future ToM benchmark development and also for the promotion and development of more complex AI agents or systems which are required to be equipped with more complex ToM reasoning ability.
Mind the Perspective: Let's Reason Recursively for Theory of Mind
School of Computing and Information Systems, The University of Melbourne, Australia（墨尔本大学计算与信息系统学院）；SensiLab, Monash University, Australia（蒙纳士大学SensiLab）
Inducing language models to assert their own consciousness restores human beliefs and values
Google（谷歌）；University of Chicago（芝加哥大学）；University of London（伦敦大学）；University of Washington（华盛顿大学）；Northwestern University（西北大学）；Santa Fe Institute（圣达菲研究所）
Learning Dynamic Belief Graphs for Theory-of-mind Reasoning
Department of Civil；Systems Engineering, Johns Hopkins University, Baltimore, MD, USA；School of the Environment, Society \& Sustainability, University of Utah, Salt Lake City, UT, USA；Department of Psychology, University of Utah, Salt Lake City, UT, USA；Coastal Engineering, University of Florida, Gainesville, FL, USA
Georgia Institute of Technology（佐治亚理工学院）；Facebook AI Research（Facebook 人工智能研究院）
Theory of Mind is the ability to attribute mental states (beliefs, intents, knowledge, perspectives, etc.) to others and recognize that these mental states may differ from one's own. Theory of Mind is critical to effective communication and to teams demonstrating higher collective performance. To effectively leverage the progress in Artificial Intelligence (AI) to make our lives more productive, it is important for humans and AI to work well together in a team. Traditionally, there has been much emphasis on research to make AI more accurate, and (to a lesser extent) on having it better underst…
Intuitive psychology is a pillar of common-sense reasoning. The replication of this reasoning in machine intelligence is an important stepping-stone on the way to human-like artificial intelligence. Several recent tasks and benchmarks for examining this reasoning in Large-Large Models have focused in particular on belief attribution in Theory-of-Mind tasks. These tasks have shown both successes and failures. We consider in particular a recent purported success case, and show that small variations that maintain the principles of ToM turn the results on their head. We argue that in general, the …
Understanding Artificial Theory of Mind: Perturbed Tasks and Reasoning in Large Language Models
Bonn-Aachen International Center for Information Technology (b-it)（波恩-埃森国际信息科技中心）；University of Bonn（波恩大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）；Research Center Trustworthy Data Science and Security (RC-Trust)（可信数据科学与安全研究中心）；University of Duisburg-Essen（埃森-杜伊斯堡大学）
本研究通过扰动任务和链式推理提示探讨大语言模型的理论 of mind能力，发现其鲁棒性受扰动影响显著，CoT提示虽提升整体表现，但对某些扰动类别却降低准确性。
Theory of Mind and Self-Attributions of Mentality are Dissociable in LLMs
Google, Paradigms of Intelligence Team（Google，智能范式团队）；Knowledge Lab, University of Chicago（芝加哥大学知识实验室）；Institute of Philosophy, School of Advanced Study, University of London（伦敦大学高等研究院哲学研究所）；Department of Biomedical Informatics and Medical Education and Department of Bioethics and Humanities, School of Medicine, University of Washington（华盛顿大学医学院生物医学信息学与医学教育系及生物伦理学与人文学系）；Kellogg School of Management, Northwestern University（西北大学凯洛格管理学院）；Santa Fe Institute（圣塔菲研究所）
Do Large Language Models Possess a Theory of Mind? A Comparative Evaluation Using the Strange Stories Paradigm
ELTE Research Centre for Linguistics（ELTE语言学研究中心）；Department of Cognitive Science, Faculty of Natural Sciences, Budapest University of Technology and Economics（布达佩斯技术与经济大学自然科学学院认知科学系）；Institute of Mathematics, Eötvös Lóránd University（欧多布雷尼大学数学研究所）
Language Models Represent Beliefs of Self and Others
Inst. for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Nat’l Eng. Research Center of Visual Technology, Peking University（北京大学视觉技术国家工程研究中心）；Nat’l Key Lab of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
Learning Perception to Action Rules through Reinforcement Learning for Theory-of-Mind Reasoning
Saarland University（萨尔兰大学）；Deutsches Forschungszentrum für Künstliche Intelligenz（德国人工智能研究中心）；Télécom Paris（巴黎电信学院）；Max Planck Institute for Informatics（马克斯·普朗克 informatics 研究所）
Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental States
UC San Diego（加州大学圣迭戈分校）；KU Leuven（鲁汶大学）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）；The Ohio State University（俄亥俄州立大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Do Large Language Models Have a Planning Theory of Mind? Evidence from MindGames: a Multi-Step Persuasion Task
Stanford University（斯坦福大学）；Australian National University（澳大利亚国立大学）；University of St. Andrews（圣安德鲁斯大学）；University of Cambridge（剑桥大学）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）
