MAgIC：基于大语言模型的多智能体在认知、适应性、理性与协作方面的研究
MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration
AI总结：
本研究针对多智能体环境下大语言模型的评估需求，提出基于竞赛的MAgIC基准框架，结合社交推理游戏与博弈论场景，通过PGM方法增强并量化评估多维度能力，揭示模型间显著差距及PGM的平均37%提升效果。
摘要：
大语言模型（LLMs）极大推动了自然语言处理的发展，展现出卓越的推理、工具使用和记忆能力。随着其应用扩展至多智能体环境，需要一个全面的评估框架来捕捉LLMs的推理、规划、协作及其他社交能力。本研究引入一种新颖的基于竞赛的基准框架，专为评估多智能体场景下的LLMs设计，提供量化指标以评估其判断、推理、欺骗、自我意识、合作、协调和理性。我们利用两个社交推理游戏和三个博弈论场景创建多样化环境。我们的框架通过概率图建模（PGM）方法增强，提升LLMs在复杂社交和认知维度中的导航能力。我们评估了七个LLMs，定量显示最强的GPT o1与最弱的Llama-2-70B之间存在超过三倍的显著能力差距。研究还证实，我们的PGM增强使所有选定模型的能力平均提升37%。我们的数据和代码可在https://github.com/cathyxl/MAgIC获取。
英文摘要：
Large Language Models (LLMs) have significantly advanced natural language processing, demonstrating exceptional reasoning, tool usage, and memory capabilities. As their applications expand into multi-agent environments, there arises a need for a comprehensive evaluation framework that captures LLMs' reasoning, planning, collaboration, and other social abilities. This work introduces a novel competition-based benchmark framework specifically designed to assess LLMs within multi-agent settings, providing quantitative metrics to evaluate their judgment, reasoning, deception, self-awareness, cooperation, coordination, and rationality. We utilize two social deduction games alongside three game-theory scenarios to create diverse environments. Our frame is fortified with the probabilistic graphic modeling (PGM) method, enhancing the LLMs' capabilities in navigating complex social and cognitive dimensions. We evaluate seven LLMs, quantitatively highlighting a significant capability gap of over threefold between the strongest, GPT o1, and the weakest, Llama-2-70B. It also confirms that our PGM enhancement boosts the abilities of all selected models by an average of 37%. Our data and code can be found here https://github.com/cathyxl/MAgIC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
