特权信息蒸馏用于语言模型
Privileged Information Distillation for Language Models
AI总结：
本文提出π-Distill和OPSD两种方法，通过特权信息蒸馏提升语言模型在复杂代理环境中的表现，优于传统监督微调加强化学习的方法。
摘要：
训练时的特权信息（PI）可以使语言模型在原本无法完成的任务上取得成功，使其成为在长期、复杂场景中强化学习的强大工具。然而，将通过PI学习的能力转移到必须在推理时没有PI的策略中仍然是一个根本性挑战。我们研究了在多轮代理环境中蒸馏前沿模型的问题，这些环境通常隐藏其内部推理过程，仅暴露动作轨迹。这打破了标准蒸馏流程，因为成功的行为是可观测的，但推理过程不是。为此，我们引入了π-Distill，一种联合教师-学生目标，通过同一模型同时训练一个PI条件教师和一个无条件学生。此外，我们还引入了On-Policy Self-Distillation（OPSD），一种通过强化学习（RL）训练的方法，使用学生与PI条件教师之间的反向KL惩罚。我们证明这两种算法都能有效蒸馏前沿代理使用仅动作的PI。具体来说，我们发现π-Distill和在某些情况下OPSD在多代理基准、模型和PI形式上优于行业标准实践（监督微调后接RL），这些实践假设可以访问完整的思维链监督。我们通过广泛的分析来补充我们的结果，该分析刻画了使PI学习有效的主要因素，主要集中在π-Distill上，并且刻画了OPSD何时具有竞争力。
英文摘要：
Training-time privileged information (PI) can enable language models to succeed on tasks they would otherwise fail, making it a powerful tool for reinforcement learning in hard, long-horizon settings. However, transferring capabilities learned with PI to policies that must act without it at inference time remains a fundamental challenge. We study this problem in the context of distilling frontier models for multi-turn agentic environments, which typically hide their internal reasoning and expose only action trajectories. This breaks standard distillation pipelines, since successful behavior is observable, but the reasoning process is not. For this, we introduce π-Distill, a joint teacher-student objective that trains a PI-conditioned teacher and an unconditioned student simultaneously using the same model. Additionally, we also introduce On-Policy Self-Distillation (OPSD), an alternative approach that trains using Reinforcement Learning (RL) with a reverse KL-penalty between the student and the PI-conditioned teacher. We show that both of these algorithms effectively distill frontier agents using action-only PI. Specifically, we find that π-Distill and, in some cases, OPSD, outperform industry standard practices (Supervised finetuning followed by RL) that assume access to full Chain-of-Thought supervision across multiple agentic benchmarks, models, and forms of PI. We complement our results with extensive analysis that characterizes the factors enabling effective learning with PI, focusing primarily on π-Distill and characterizing when OPSD is competitive.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
