arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.00746cs.AI
收藏

解读数字侦探:理解大型语言模型在多智能体悬疑游戏中的行为与能力

Deciphering Digital Detectives: Understanding LLM Behaviors and Capabilities in Multi-Agent Mystery Games

Université de Montréal(蒙特利尔大学);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所)

Dekun Wu, Haochen Shi, Zhiyuan Sun, Bang Liu

AI总结:

本文面向剧本杀构建首个数据集与多智能体 LLM 交互框架,并通过案件掌握、推理评估及上下文学习提升智能体表现,建立新基准。

摘要:

在本研究中,我们探索了大型语言模型(LLM)在剧本杀(Jubensha)中的应用;剧本杀是一种中国侦探角色扮演游戏,也是人工智能(AI)驱动游戏中的一个新兴领域。我们介绍了首个专门面向剧本杀的数据集,其中包含角色剧本和游戏规则,以促进在这一复杂叙事环境中的 AI 智能体开发。我们的工作还提出了一个使用 LLM 的独特多智能体交互框架,使 AI 智能体能够自主参与该游戏。为评估这些 AI 智能体的游戏表现,我们开发了新的方法,衡量它们对案件信息的掌握程度和推理技能。此外,我们结合了上下文学习(in-context learning)的最新进展,以提升智能体在信息收集、凶手识别和逻辑推理方面的表现。实验结果验证了我们所提方法的有效性。本工作旨在为理解 LLM 能力提供新的视角,并为评估基于大型语言模型的智能体建立新的基准。

英文摘要:

In this study, we explore the application of Large Language Models (LLMs) in \textit{Jubensha}, a Chinese detective role-playing game and a novel area in Artificial Intelligence (AI) driven gaming. We introduce the first dataset specifically for Jubensha, including character scripts and game rules, to foster AI agent development in this complex narrative environment. Our work also presents a unique multi-agent interaction framework using LLMs, allowing AI agents to autonomously engage in this game. To evaluate the gaming performance of these AI agents, we developed novel methods measuring their mastery of case information and reasoning skills. Furthermore, we incorporated the latest advancements in in-context learning to improve the agents' performance in information gathering, murderer identification, and logical reasoning. The experimental results validate the effectiveness of our proposed methods. This work aims to offer a novel perspective on understanding LLM capabilities and establish a new benchmark for evaluating large language model-based agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑