arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.04686cs.AI
收藏

在狼人杀游戏中通过迭代潜在空间策略优化学习战略语言智能体

Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization

Tsinghua University, Beijing, China(清华大学);Beijing Zhongguancun Academy, Beijing, China(北京中关村学院);Shanghai Qi Zhi Institute, Shanghai, China(上海启智研究所)

Zelai Xu, Wanjun Gu, Chao Yu, Yi Wu, Yu Wang

AI总结:

针对LLM智能体在战略语言游戏中的偏差与探索不足,提出LSPO迭代框架,结合博弈论与DPO微调,在狼人杀中提升策略与语言能力并超越现有智能体。

摘要:

大型语言模型(LLM)智能体最近在开放域对话和多步决策等各个领域展现了令人印象深刻的能力。然而,这些智能体在解决诸如狼人杀等战略语言游戏时仍面临挑战,这类游戏既需要战略决策,又需要自由形式的语言交互。现有的LLM智能体常常在动作分布上存在内在偏差,并且对无界文本动作空间的探索有限,导致性能次优。为应对这些挑战,我们提出了潜在空间策略优化(LSPO),这是一个将博弈论方法与LLM微调相结合的迭代框架,用于构建战略语言智能体。LSPO利用了这样一个观察:虽然语言空间在组合上是巨大的,但底层策略空间相对紧凑。我们首先将自由形式的表述映射到有限的潜在策略空间,得到一个抽象化的扩展形式博弈。然后,我们应用诸如反事实遗憾最小化(CFR)等博弈论方法在潜在空间中优化策略。最后,我们通过直接偏好优化(DPO)微调LLM,使其与学习到的策略对齐。通过在这些步骤之间迭代交替,我们的LSPO智能体逐步增强了战略推理和语言交流能力。在狼人杀游戏上的实验表明,我们的智能体迭代扩展策略空间并提升性能,且优于现有的狼人杀智能体,突显了它们在具有战略互动的自由形式语言游戏中的有效性。

英文摘要:

Large language model (LLM) agents have recently demonstrated impressive capabilities in various domains like open-ended conversation and multi-step decision-making. However, it remains challenging for these agents to solve strategic language games, such as Werewolf, which demand both strategic decision-making and free-form language interactions. Existing LLM agents often suffer from intrinsic bias in their action distributions and limited exploration of the unbounded text action space, resulting in suboptimal performance. To address these challenges, we propose Latent Space Policy Optimization (LSPO), an iterative framework that combines game-theoretic methods with LLM fine-tuning to build strategic language agents. LSPO leverages the observation that while the language space is combinatorially large, the underlying strategy space is relatively compact. We first map free-form utterances into a finite latent strategy space, yielding an abstracted extensive-form game. Then we apply game-theoretic methods like Counterfactual Regret Minimization (CFR) to optimize the policy in the latent space. Finally, we fine-tune the LLM via Direct Preference Optimization (DPO) to align with the learned policy. By iteratively alternating between these steps, our LSPO agents progressively enhance both strategic reasoning and language communication. Experiment on the Werewolf game shows that our agents iteratively expand the strategy space with improving performance and outperform existing Werewolf agents, underscoring their effectiveness in free-form language games with strategic interactions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑