发表机构
Tsinghua University; Zhejiang University; Alibaba Group（清华大学; 浙江大学; 阿里巴巴集团）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出Self-OPD，一种无需教师模型的流匹配模型在线策略蒸馏框架，通过分支SDE候选与奖励比较优化速度场，在基准测试中优于现有无教师的RL和OPD方法。
AI 中文摘要
在线策略蒸馏（OPD）利用预训练的专用教师模型提供密集监督信号，已在大语言模型（LLMs）中取得显著成功，且最近被应用于流匹配模型。然而，该范式存在两个主要问题：其一，为每个新目标训练单独的特定任务教师会产生高计算成本；其二，教师与学生分布之间的差异常导致生成轨迹上的复合误差。本文提出Self-OPD，一种用于流匹配模型的无教师OPD框架，将学生自身的自探索转化为逐步监督。在每个时间步，Self-OPD将确定性的下一状态预测分支为K个随机SDE候选，用ODE采样器展开它们，并将其奖励与确定性自参考基线比较以获得归一化优势。速度场通过全分支推拉目标优化，其中高优势分支吸引学生，低优势分支在方向感知衰减和SDE方差归一化下排斥学生。对于多目标对齐，Self-OPD在奖励层面融合归一化分数，避免直接梯度冲突。在单奖励和混合奖励基准上的实验表明，Self-OPD在无需特定任务教师的情况下优于现有RL和OPD方法。
英文摘要
On-policy distillation (OPD), which leverages a pre-trained, specialized teacher model to provide dense supervisory signals, has achieved significant success in Large Language Models (LLMs) and has recently been adapted to flow matching models. However, this paradigm suffers from two major issues: First, training a separate, task-specific teacher for every new objective incurs high computational costs. Second, the discrepancy between teacher and student distributions often leads to compounding errors along the generation trajectory. In this paper, we introduce \textbf{Self-OPD}, a teacher-free OPD framework for flow matching models that turns the student's own self-exploration into step-wise supervision. At each timestep, Self-OPD branches the deterministic next-state prediction into $K$ stochastic SDE candidates, rolls them out with the ODE sampler, and compares their rewards against a deterministic self-reference baseline to obtain normalized advantages. The velocity field is optimized with an all-branch pull-push objective, where high-advantage branches attract the student and low-advantage branches repel it under direction-aware attenuation and SDE-variance normalization. For multi-objective alignment, Self-OPD fuses normalized scores at the reward level, avoiding direct gradient conflict. Experiments on single and mixed reward benchmarks show that Self-OPD outperforms prior RL and OPD methods without task-specific teachers.
Comments19 pages, 10 figures
