一个学生，多个教师：通过软提示特权上下文进行多任务在线策略蒸馏
One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context
浏览论文内容
中文总结 AI 辅助
研究提出一种多任务在线策略蒸馏方法，教师与学生仅通过可学习软提示区分，在骨干冻结时训练特定任务教师。单任务变体训练参数少且效果好，多任务变体保持通用能力基准同时实现最佳总体平均水平。
中文摘要 AI 辅助
在线策略自蒸馏（OPSD）通过共享学生骨干并监督自身展开的教师来教授大语言模型新技能。现有教师要么在输入时注入特权上下文——引发事后合理化，要么微调权重，在跨任务中积累偏差和遗忘。我们提出了一种方法，其教师与学生的区别仅在于一个可学习的软提示：在骨干冻结的情况下基于（x，y_gold）对进行训练，该提示产生一个特定任务的教师，保留学生的确切表示几何形状。该方法通过将合并语料库中的每个示例路由到其相应的软提示教师自然地扩展到多任务设置，允许单个学生并行从K个教师那里吸收知识；在推理时，所有提示都被丢弃。在Qwen3 - 1.7B - Base和Phi - 4 - mini - instruct上进行四个任务（科学、工具使用、生物学、数学）的实验，单任务变体（带有PT教师的OPD）在训练参数数量少几个数量级的情况下匹配或超过完全微调，多任务变体在保持通用能力基准的同时实现了最佳总体平均水平（Qwen3 - 1.7B - Base上为56.2），而顺序SFT会使两者都下降。
英文摘要
On-policy self-distillation (OPSD) teaches large language models new skills through a teacher that shares the student's backbone and supervises its own rollouts. Existing teachers either inject privileged context at the input -- inducing post-hoc rationalization -- or fine-tune weights, accumulating drift and forgetting across tasks. We propose \method, whose teacher differs from the student only by a learnable soft prompt: trained on $(x, y_\text{gold})$ pairs with the backbone frozen, the prompt yields a task-specific teacher that preserves the student's exact representational geometry. \method\ extends naturally to multi-task settings by routing each example in a merged corpus to its corresponding soft-prompt teacher, allowing a single student to absorb knowledge from $K$ teachers in parallel; at inference, all prompts are discarded. On Qwen3-1.7B-Base and Phi-4-mini-instruct across four tasks (Science, Tool Use, Biology, Math), the single-task variant (OPD with a PT teacher) matches or exceeds full fine-tuning while training orders of magnitude fewer parameters, and the multi-task variant achieves the best overall average ($56.2$ on Qwen3-1.7B-Base) while preserving general-capability benchmarks -- in contrast to sequential SFT, which degrades both.
发表机构
- University of Wisconsin–Madison（威斯康星大学麦迪逊分校）
- Johns Hopkins University（约翰霍普金斯大学）
- Nanyang Technological University（南洋理工大学）
机构由 AI 辅助整理，请以论文原文为准。
