多任务智能强化学习的熵步长策略优化
Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learning
AI总结：
研究多任务智能强化学习中任务间探索-利用步长不匹配问题，提出熵步长策略优化（EPPO），核心是任务级动态裁剪机制，实验证明EPPO在多任务智能基准上结果优于同类方法。
摘要：
强化学习（RL）的最新突破凸显了其在复杂智能大语言模型（LLM）任务中的潜力。然而，现有工作主要集中在单任务设置上，而实际部署需要一个能够同时解决多个任务的通才智能体。在这项工作中，我们发现了多任务智能RL中一个关键但未被充分探索的现象：不同任务可能表现出探索-利用步长不匹配。具体而言，较简单的任务可能过早收敛到低熵策略，阻碍较难任务的学习，而较难任务又会将较简单任务推回高熵探索。这种来回交互会产生任务间的熵交叉和频繁的熵尖峰。受此观察启发，我们为多任务智能LLM引入了熵步长策略优化（EPPO），它协调跨任务的熵以稳定多任务优化。EPPO的核心是一个任务级动态裁剪机制，用任务熵感知自适应边界取代组相对策略优化（GRPO）中的固定裁剪阈值，对过度自信的任务收紧更新，对探索不足的任务放松更新。在多任务智能基准上的实验表明，所提出的EPPO产生的结果优于同类方法。
英文摘要：
Recent breakthroughs of Reinforcement Learning (RL) have highlighted its potential for complex agentic Large Language Model (LLM) tasks. However, existing efforts largely focus on single-task settings, whereas real-world deployment necessitates a generalist agent capable of solving multiple tasks simultaneously. In this work, we identify a critical yet underexplored phenomenon in multi-task agentic RL: different tasks can exhibit exploration-exploitation pace mismatch. Specifically, easier tasks may converge early to low-entropy policies that hinder learning on harder tasks, while harder tasks can, in turn, push easier tasks back toward high-entropy exploration. This back-and-forth interaction creates inter-task entropy crossovers and frequent entropy spikes. Inspired by this observation, we introduce Entropy Pacing Policy Optimization (EPPO) for multi-task agentic LLMs, which coordinates entropy across tasks to stabilize multi-task optimization. At the core of EPPO is a task-wise dynamic clipping mechanism that replaces the fixed clipping threshold in Group Relative Policy Optimization (GRPO) with a task entropy-aware adaptive bound, tightening updates for over-confident tasks while relaxing them for under-explored ones. Experiments on the multi-task agentic benchmarks demonstrate that the proposed EPPO yields results superior to its counterparts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
