PolicyEvolve：通过基于种群的训练利用LLMs演化多玩家游戏中的程序化策略
PolicyEvolve: Evolving Programmatic Policies by LLMs for multi-player games via Population-Based Training
AI总结：
PolicyEvolve利用LLMs通过基于种群的训练框架，为多玩家游戏自动生成可解释的程序化策略，减少人工代码依赖和环境交互，实现高性能对抗策略。
摘要：
多智能体强化学习（MARL）通过自博弈在解决复杂多玩家游戏方面取得了显著进展。然而，训练有效的对抗性策略需要数百万个经验样本和大量计算资源。此外，这些策略缺乏可解释性，阻碍了其实际部署。近期，研究人员成功利用大型语言模型（LLMs）为单智能体任务生成程序化策略，将基于神经网络的策略转化为具有高执行效率的可解释规则代码。受此启发，我们提出PolicyEvolve，一个用于在多玩家游戏中生成程序化策略的通用框架。PolicyEvolve显著减少了对人工设计策略代码的依赖，以最少的环境交互实现高性能策略。该框架包含四个模块：全局池（Global Pool）、局部池（Local Pool）、策略规划器（Policy Planner）和轨迹评论器（Trajectory Critic）。全局池保留迭代训练过程中积累的精英策略。局部池存储当前迭代的临时策略；只有该池中性能足够高的策略才会被提升至全局池。策略规划器作为核心策略生成模块，从全局池中采样前三个策略，基于环境信息生成当前迭代的初始策略，并利用轨迹评论器的反馈对该策略进行改进。改进后的策略随后被存入局部池。此迭代过程持续进行，直到该策略对全局池的平均胜率达到足够高，此时将其整合进全局池。轨迹评论器分析当前策略的交互数据，识别漏洞并提出方向性改进建议，以指导策略规划器。
英文摘要：
Multi-agent reinforcement learning (MARL) has achieved significant progress in solving complex multi-player games through self-play. However, training effective adversarial policies requires millions of experience samples and substantial computational resources. Moreover, these policies lack interpretability, hindering their practical deployment. Recently, researchers have successfully leveraged Large Language Models (LLMs) to generate programmatic policies for single-agent tasks, transforming neural network-based policies into interpretable rule-based code with high execution efficiency. Inspired by this, we propose PolicyEvolve, a general framework for generating programmatic policies in multi-player games. PolicyEvolve significantly reduces reliance on manually crafted policy code, achieving high-performance policies with minimal environmental interactions. The framework comprises four modules: Global Pool, Local Pool, Policy Planner, and Trajectory Critic. The Global Pool preserves elite policies accumulated during iterative training. The Local Pool stores temporary policies for the current iteration; only sufficiently high-performing policies from this pool are promoted to the Global Pool. The Policy Planner serves as the core policy generation module. It samples the top three policies from the Global Pool, generates an initial policy for the current iteration based on environmental information, and refines this policy using feedback from the Trajectory Critic. Refined policies are then deposited into the Local Pool. This iterative process continues until the policy achieves a sufficiently high average win rate against the Global Pool, at which point it is integrated into the Global Pool. The Trajectory Critic analyzes interaction data from the current policy, identifies vulnerabilities, and proposes directional improvements to guide the Policy Planner
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
