arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.27140cs.AI
收藏

StepOPSD: 面向智能体强化学习的步骤感知在线偏好蒸馏

StepOPSD: Step-Aware Online Preference Self-Distillation for Agent Reinforcement Learning

Independent Researcher(独立研究者);Tencent(腾讯);DeepWisdom(深智沃)

Yanfei Zhang, Xu Lin, Chenglin Wu

AI总结:

提出StepOPSD框架,以智能体步骤为信用分配单元,通过事后增强教师上下文重新评分步骤段,并在GRPO更新前进行归一化每步信用预算的优势塑造,解决多轮智能体强化学习中的信用分配不匹配问题。

摘要:

多轮智能体的强化学习存在信用分配不匹配问题:奖励稀疏且基于轨迹,而成功往往取决于少数局部决策。现有的在线策略蒸馏(OPD)提供了更密集的令牌级监督,但通常将异质的智能体轨迹视为整体字符串而非因果交互单元。我们提出StepOPSD,一种事后回放偏好自蒸馏框架,以智能体步骤作为信用重分配的单位。StepOPSD将轨迹分解为以动作中心的步骤段,在事后增强的教师上下文中重新评分,并将令牌级对数概率差距转化为符号保持的优势塑造,在GRPO更新前进行归一化的每步信用预算。在ALFWorld和Search-QA上使用Qwen3-1.7B和Qwen2.5-3B-Instruct的实验中,StepOPSD在对局部因果错误最敏感的子集上取得了最佳或次佳结果,包括ALFWorld Heat(79.1%)、PickTwo(95.0%)、Search-QA TriviaQA(61.6%)的第一名,以及HotpotQA(40.4%)的并列最佳。结果进一步揭示了一致的双旋钮定律:较小的α_clip作为广泛稳定的局部信任区域,而最优全局混合强度λ_mix依赖于任务。这些发现表明,当轨迹级奖励与决定下游成功的局部动作弱对齐时,步骤感知蒸馏最为有用。

英文摘要:

Reinforcement learning for multi-turn agents suffers from a credit-assignment mismatch: rewards are sparse and trajectory-level, while success often hinges on a few local decisions. Existing online policy distillation (OPD) provides denser token-level supervision, but typically treats heterogeneous agent trajectories as monolithic strings rather than causal interaction units. We present StepOPSD, a post-rollout preference self-distillation framework that takes the agent step as the unit of credit redistribution. StepOPSD decomposes trajectories into action-centered step segments, rescoring them under hindsight-enriched teacher contexts and converting token-level log-probability gaps into sign-preserving advantage shaping with a normalized per-step credit budget before the GRPO update. Across ALFWorld and Search-QA with Qwen3-1.7B and Qwen2.5-3B-Instruct, StepOPSD attains best or second-best results on subsets most sensitive to local causal errors, including first-place performance on ALFWorld Heat (79.1%), PickTwo (95.0%), Search-QA TriviaQA (61.6%), and tied-best performance on HotpotQA (40.4%). The results further reveal a consistent two-knob law: smaller α_clip acts as a broadly stabilizing local trust region, whereas the optimal global mixing strength λ_mix remains task-dependent. These findings suggest that step-aware distillation is most useful when trajectory-level rewards are weakly aligned with the local action that determines downstream success.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑