ToMATO：将角色扮演大语言模型的心理状态语言化以用于心理理论基准测试
ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind
AI总结：
针对现有心理理论基准的局限，该研究提出基于信息不对称对话生成的ToMATO基准，覆盖多类心理状态与人格特质，评测显示主流LLM仍落后于人类水平。
摘要：
现有的心理理论（Theory of Mind，ToM）基准与真实场景存在三方面差异：1）仅评估信念等有限范围的心理状态；2）未全面探究错误信念；3）忽略了角色的多样人格特质。为解决这些问题，我们提出了ToMATO——一个以对话类多项选择题形式构建的新型心理理论基准。\nToMATO通过存在信息不对称的大语言模型（LLM）间对话生成。我们采用一种提示方法，要求进行角色扮演的LLM在每次发言前将自身想法语言化，以此捕捉信念、意图、愿望、情绪、知识五大类的一阶和二阶心理状态。这些被语言化的想法可作为评估对话中角色心理状态的问题的答案。此外，通过向其他角色隐藏想法引入的信息不对称，会促使各类心理状态相关的错误信念产生。为LLM分配不同的人格特质，进一步丰富了发言内容和想法的多样性。\nToMATO包含5400个问题、753段对话和15种人格特质模式。我们的分析显示，这种数据集构建方法因角色扮演LLM间的信息不对称会频繁生成错误信念，且能有效体现多样人格。我们在ToMATO上评估了9种LLM，发现即使是GPT-4o mini也落后于人类表现，尤其是在理解错误信念方面，且对不同人格特质缺乏鲁棒性。
英文摘要：
Existing Theory of Mind (ToM) benchmarks diverge from real-world scenarios in three aspects: 1) they assess a limited range of mental states such as beliefs, 2) false beliefs are not comprehensively explored, and 3) the diverse personality traits of characters are overlooked. To address these challenges, we introduce ToMATO, a new ToM benchmark formulated as multiple-choice QA over conversations. ToMATO is generated via LLM-LLM conversations featuring information asymmetry. By employing a prompting method that requires role-playing LLMs to verbalize their thoughts before each utterance, we capture both first- and second-order mental states across five categories: belief, intention, desire, emotion, and knowledge. These verbalized thoughts serve as answers to questions designed to assess the mental states of characters within conversations. Furthermore, the information asymmetry introduced by hiding thoughts from others induces the generation of false beliefs about various mental states. Assigning distinct personality traits to LLMs further diversifies both utterances and thoughts. ToMATO consists of 5.4k questions, 753 conversations, and 15 personality trait patterns. Our analysis shows that this dataset construction approach frequently generates false beliefs due to the information asymmetry between role-playing LLMs, and effectively reflects diverse personalities. We evaluate nine LLMs on ToMATO and find that even GPT-4o mini lags behind human performance, especially in understanding false beliefs, and lacks robustness to various personality traits.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
