PADD: 面向非路由器教师指导MoE学生学习的路径对齐解压缩蒸馏
PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning
AI总结：
提出路径对齐解压缩蒸馏（PADD）框架，通过四阶段两阶段流程将密集教师知识蒸馏到混合专家（MoE）学生中，同时学习高质量路由策略，在数学推理任务上显著优于基线。
摘要：
随着大型语言模型（LLMs）持续扩展，在固定计算预算下增长模型容量变得越来越具有挑战性。我们提出路径对齐解压缩蒸馏（PADD），这是一个将知识从无显式路由的密集教师蒸馏到混合专家（MoE）学生中，同时学习高质量路由策略的框架。PADD将知识蒸馏组织为两个阶段的四个阶段：初始化阶段（阶段I）通过教师神经元聚类和学生专家预热在学生专家中构建多样功能，以及训练阶段（阶段II–IV）将在线自适应蒸馏、路径细化策略优化和奖励增强负载平衡集成在单一训练流程中。在数学推理基准上的实验表明，在相同推理成本下，PADD相比强基线取得了显著提升，且MoE学生能够匹配或超越其密集教师。实验还展示了有效的教师到学生知识蒸馏和稳定的路由行为。
英文摘要：
As large language models (LLMs) continue to scale, it becomes increasingly challenging to grow model capacity under fixed computation budgets. We propose Path-Aligned Decompression Distillation (PADD), a framework for distilling knowledge from dense teachers without explicit routing into mixture-of-experts (MoE) students while learning high-quality routing policies. PADD organizes knowledge distillation into four stages in two phases: an initialization phase (Stage I) that builds diverse functionality in the student's experts through teacher neuron clustering and student-expert warmup, and a training phase (Stages II--IV) that integrates online adaptive distillation, path-refined policy optimization, and reward-augmented load balancing in a single training pipeline. Experiments on mathematical reasoning benchmarks demonstrate that PADD yields substantial gains over strong baselines at the same inference cost and that the MoE student can match or surpass its dense teacher. They also demonstrate effective teacher-to-student knowledge distillation and stable routing behavior.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
