弱批评者造就强学习者：用于可扩展监督的在策略批评蒸馏
Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight
AI总结：
提出在策略批评蒸馏（OPCD）方法，利用弱模型作为批评者提供修订方向，通过自适应自教师信号蒸馏批评引导的行为，提升强模型在推理和对齐基准上的表现。
摘要：
随着大型语言模型变得更强，弱监督者可能无法为复杂输出提供可靠的标签、偏好或最终判断，限制了弱到强的泛化和可扩展监督。我们研究了一种更易处理的弱监督形式：使用弱模型作为批评者，而不是作为标注者或评判者。弱批评者不需要解决任务或选择正确答案，只需提供非误导性的修订方向，帮助强模型更好地利用自身知识。我们将这种设置称为*弱批评者强监督*。我们首先表明，弱批评可以在推理时改进冻结的强模型，并且批评质量是这种改进的关键。然后，我们提出渐进式在策略批评蒸馏（**OPCD**），它过滤高质量的批评，并通过自适应自教师信号将批评引导的行为蒸馏到强模型中。在推理和对齐基准上的实验表明，我们的方法在训练轮次中改进了强模型，为使用弱监督的可扩展监督提供了一条有效路径。
英文摘要：
As large language models become stronger, weak supervisors may fail to provide reliable labels, preferences, or final judgments for complex outputs, limiting both weak-to-strong generalization and scalable oversight. We study a more tractable form of weak supervision: using a weak model as a critic rather than as a labeler or judge. Instead of solving the task or selecting the correct answer, the weak critic only needs to provide a non-misleading revision direction that helps the strong model better use its own knowledge. We call this setting *weak-critic strong oversight*. We first show that weak critiques can improve frozen strong models at inference time, and that critique quality is key to this improvement. We then propose progressive on-policy critique distillation (**OPCD**), which filters high-quality critiques and distills critic-guided behavior into the strong model through adaptive self-teacher signals. Experiments on reasoning and alignment benchmarks show that our method improves strong models over training epochs, suggesting an effective path for scalable oversight with weak supervision.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
