跨行为策略优化：通过共享回归目标从离线演示及在线、离线策略 rollout 中进行统一学习
Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target
AI总结：
针对推理后训练中离线演示与策略 rollout 目标分离的问题，提出 CBPO 方法统一多数据源，在数学推理基准上实现领先性能，Qwen3-1.7B 版本较 SFT DAPO 有显著提升。
摘要：
推理后训练依赖于离线演示和策略生成的 rollout，但传统的监督微调（SFT）后接强化学习（RL）的方式将两者目标分离，限制了对收集到的响应的持续复用。我们提出跨行为策略优化（Cross-Behavior Policy Optimization, CBPO），其通过固定参考目标使这些数据源能与回归损失兼容。由于生成策略不进入该目标，正、负离线完成结果、参考响应以及当前和历史 rollout 可共享通用的奖励标记记录格式和回放缓冲区。随着新响应进入缓冲区，优化过程保持相同的回归目标，无需协调独立的 SFT 和 RL 目标，也无需存储行为策略概率、重要性权重或学习到的评论者。限制可用数据源可得到离线、在线策略、离线策略变体，而 CBPO-Unified 则将它们结合。在两种模型规模和四个数学推理基准上，CBPO 在各基准和数据 regime 中均实现领先的 avg@32 和 pass@32 性能。在 Qwen3-1.7B 上，CBPO-Unified 相比 SFT DAPO 将基准平均 avg@32 和 pass@32 分别提升 2.33 和 4.44 个百分点；在回放 ablation 中，保留历史 rollout 相比无历史配置将这些指标分别提升 1.87 和 3.72 个百分点。
英文摘要：
Reasoning post-training draws on offline demonstrations and policy-generated rollouts, yet conventional supervised fine-tuning (SFT) followed by reinforcement learning (RL) separates their objectives and limits continued reuse of collected responses. We propose \textbf{Cross-Behavior Policy Optimization} (CBPO), which makes these sources compatible with one regression loss through a fixed-reference target. Because the generating policy does not enter this target, positive and negative offline completions, reference responses, and current and historical rollouts can share a common reward-labeled record format and replay buffer. As new responses enter the buffer, optimization retains the same regression objective without coordinating separate SFT and RL objectives or requiring stored behavior-policy probabilities, importance weighting, or a learned critic. Restricting the available sources gives offline, on-policy, and off-policy variants, while CBPO-Unified combines them. Across two model scales and four mathematical reasoning benchmarks, CBPO delivers leading \texttt{avg@32} and \texttt{pass@32} performance across benchmarks and data regimes. On Qwen3-1.7B, CBPO-Unified improves benchmark-averaged \texttt{avg@32} and \texttt{pass@32} over SFT DAPO by 2.33 and 4.44 percentage points, respectively. In the replay ablation, retaining historical rollouts improves these metrics by 1.87 and 3.72 percentage points over the no-history configuration.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
