LLM-Hanabi：在不完全信息协作博弈中评估具备心智理论与推理推断能力的多智能体博弈表现
LLM-Hanabi: Evaluating Multi-Agent Gameplays with Theory-of-Mind and Rationale Inference in Imperfect Information Collaboration Game
AI总结：
本研究提出新型基准LLM-Hanabi，基于合作博弈Hanabi评估大语言模型的行为逻辑推断与心智理论能力，发现一阶心智理论对协作表现的影响强于高阶推理，为提升多智能体协作能力指明方向。
摘要：
有效的多智能体协作需要智能体推断他人行动背后的行为逻辑，这一能力根植于心智理论（Theory-of-Mind, ToM）。尽管近期的大语言模型（Large Language Models, LLMs）擅长逻辑推理，但它们在动态协作场景中推断行为逻辑的能力仍未得到充分探究。本研究提出了LLM-Hanabi这一新型基准，采用合作类游戏《Hanabi》来评估LLMs的行为逻辑推断能力与心智理论水平。我们的框架配备了一套自动化评估系统，可同时衡量博弈表现与ToM能力水平。在多个模型的测试中，我们发现ToM能力与博弈内成功率存在显著的正相关关系。值得注意的是，一阶ToM（解读他人意图）与表现的相关性强于二阶ToM（预测他人的解读）。这些发现表明，要实现有效的AI协作，准确解读合作伙伴行为逻辑的能力比高阶推理更为关键。我们得出结论：优先发展一阶ToM是提升未来模型协作能力的一个极具前景的方向。
英文摘要：
Effective multi-agent collaboration requires agents to infer the rationale behind others' actions, a capability rooted in Theory-of-Mind (ToM). While recent Large Language Models (LLMs) excel at logical inference, their ability to infer rationale in dynamic, collaborative settings remains under-explored. This study introduces LLM-Hanabi, a novel benchmark that uses the cooperative game Hanabi to evaluate the rationale inference and ToM of LLMs. Our framework features an automated evaluation system that measures both game performance and ToM proficiency. Across a range of models, we find a significant positive correlation between ToM and in-game success. Notably, first-order ToM (interpreting others' intent) correlates more strongly with performance than second-order ToM (predicting others' interpretations). These findings highlight that for effective AI collaboration, the ability to accurately interpret a partner's rationale is more critical than higher-order reasoning. We conclude that prioritizing first-order ToM is a promising direction for enhancing the collaborative capabilities of future models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
