EPO:通过强化学习实现LLM战略推理的显式策略优化
EPO: Explicit Policy Optimization for Strategic Reasoning in LLMs via Reinforcement Learning
AI总结:
针对LLM在复杂现实场景中战略推理能力不足的问题,提出显式策略优化(EPO),通过多轮强化学习与迭代自博弈训练可插入任意智能体的战略推理模型,显著提升了长期目标对齐及策略生成能力。
摘要:
大型语言模型(LLMs)在数学和编程等具有明确解的良定义问题上展现了出色的推理能力。然而,它们在商业谈判等复杂现实场景中仍面临困难,这类场景需要战略推理——一种在不确定性中驾驭动态环境并协调长期目标的能力。现有的战略推理方法在适应性、可扩展性及向新上下文迁移策略方面面临挑战。为解决这些问题,我们提出了用于战略推理的显式策略优化(EPO),其特色是一个在开放式动作空间中提供策略的LLM,且能被插入任意LLM智能体以激发目标导向行为。为提升适应性和策略可迁移性,我们通过多轮强化学习(RL)训练该战略推理模型,利用过程奖励和迭代自博弈。跨社会和物理领域的实验表明,EPO能通过增强的战略推理实现长期目标对齐,在社会对话和网页导航任务上取得了最先进的性能。我们的发现揭示了EPO中涌现的各种协作推理机制及其在生成新颖策略上的有效性,凸显了其在现实应用中战略推理的潜力。代码和数据可在 https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/EPO 获取。
英文摘要:
Large Language Models (LLMs) have shown impressive reasoning capabilities in well-defined problems with clear solutions, such as mathematics and coding. However, they still struggle with complex real-world scenarios like business negotiations, which require strategic reasoning-an ability to navigate dynamic environments and align long-term goals amidst uncertainty. Existing methods for strategic reasoning face challenges in adaptability, scalability, and transferring strategies to new contexts. To address these issues, we propose explicit policy optimization (EPO) for strategic reasoning, featuring an LLM that provides strategies in open-ended action space and can be plugged into arbitrary LLM agents to motivate goal-directed behavior. To improve adaptability and policy transferability, we train the strategic reasoning model via multi-turn reinforcement learning (RL),utilizing process rewards and iterative self-play. Experiments across social and physical domains demonstrate EPO's ability of long-term goal alignment through enhanced strategic reasoning, achieving state-of-the-art performance on social dialogue and web navigation tasks. Our findings reveal various collaborative reasoning mechanisms emergent in EPO and its effectiveness in generating novel strategies, underscoring its potential for strategic reasoning in real-world applications. Code and data are available at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/EPO.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。