OpenToM：一个用于评估大型语言模型心智理论推理能力的综合基准
OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models
AI总结：
针对现有心智理论基准的不足，本文提出OpenToM基准，通过更长叙事、明确性格、意图驱动行动及物理心理双世界问题全面评估大型语言模型，发现其在物理世界建模强而心理世界追踪弱。
摘要：
神经心智理论（N-ToM），即机器理解并追踪他人心理状态的能力，在开发具备社会智能的智能体中至关重要。然而，现有的 N-ToM 基准存在诸多不足，包括含有模糊且人工编造的叙事、缺乏人物性格特征与偏好、缺少针对角色心理状态的问题，以及所提问题多样性有限。针对这些问题，我们构建了 OpenToM，一个用于评估 N-ToM 的新基准，其具备：(1) 更长且更清晰的叙事故事，(2) 具有明确性格特征的角色，(3) 由角色意图触发的行动，以及 (4) 旨在挑战 LLM 对物理和心理世界角色心理状态建模能力的问题。使用 OpenToM，我们发现最先进的 LLM 在建模物理世界心理状态的某些方面表现出色，但在追踪心理世界角色的心理状态时存在不足。
英文摘要：
Neural Theory-of-Mind (N-ToM), machine's ability to understand and keep track of the mental states of others, is pivotal in developing socially intelligent agents. However, prevalent N-ToM benchmarks have several shortcomings, including the presence of ambiguous and artificial narratives, absence of personality traits and preferences, a lack of questions addressing characters' psychological mental states, and limited diversity in the questions posed. In response to these issues, we construct OpenToM, a new benchmark for assessing N-ToM with (1) longer and clearer narrative stories, (2) characters with explicit personality traits, (3) actions that are triggered by character intentions, and (4) questions designed to challenge LLMs' capabilities of modeling characters' mental states of both the physical and psychological world. Using OpenToM, we reveal that state-of-the-art LLMs thrive at modeling certain aspects of mental states in the physical world but fall short when tracking characters' mental states in the psychological world.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
