AI 中文总结
研究混合专家架构路由逻辑，揭示其是哈夫曼编码体现，提出频率多样性定律。发现Qwen3.5-35B-A3B的冗余陷阱，提出子集差异剪枝策略，证明可释放模型哈夫曼效率，建议下一代模型转向最小描述长度最优。
AI 中文摘要
混合专家架构彻底改变了模型扩展方式，但其路由的底层逻辑仍是个黑箱。本文揭示了一个基本的主导原则：混合专家路由不仅是选择，更是哈夫曼编码的一种体现。我们引入频率多样性定律，表明像Phi-3.5-MoE和Gemma-4-27B-A4B等模型能自发地充当信息理论引擎，为常见令牌分配稀疏专家资源，为思维链轨迹中罕见、复杂任务调用高多样性专家委员会。然而，我们发现Qwen3.5-35B-A3B存在关键冗余陷阱，当有效稀疏度足够低时，负载均衡会无意中造成功能冗余，掩盖哈夫曼效率信号。为弥合这一差距，我们提出子集差异剪枝，一种消除功能重复的策略。实验表明剪枝不会降低推理能力，反而能释放模型潜在的哈夫曼效率，使逻辑简化为高效路径。我们的发现表明下一代混合专家模型应从强制负载均衡转向最小描述长度最优，为高频信息分配更短专家路由码，为低频信息分配更长、更多样化码，将路由从启发式转变为有原则的压缩引擎。
英文摘要
Mixture-of-Experts architectures have revolutionized scaling, yet the underlying logic of their routing remains a black box. In this paper, we uncover a fundamental governing principle: MoE routing is not merely selection, but a manifestation of Huffman Coding. We introduce the Frequency-Diversity Law, revealing that state-of-the-art models, such as Phi-3.5-MoE and Gemma-4-27B-A4B, spontaneously act as information-theoretic engines. These models allocate sparse expert resources for common tokens while invoking high-diversity expert committees for rare, complex tasks found in chain-of-thought trajectories. However, we identify a critical redundancy trap in Qwen3.5-35B-A3B: when effective sparsity (k/E_eff) is sufficiently low, load-balancing inadvertently imposes functional redundancy, masking the underlying Huffman efficiency signal. To bridge this gap, we propose Subset Difference Pruning, a surgical strategy to eliminate functional duplicates. We demonstrate that pruning does not degrade reasoning; instead, it unleashes the model's latent Huffman efficiency, forcing the logic to collapse into streamlined, high-density paths. Our findings suggest that the next generation of MoEs should move beyond forced load-balancing toward Minimum Description Length (MDL) optimality, assigning shorter expert-routing codes to high-frequency information and longer, more diverse codes to low-frequency information, thereby transforming routing from a heuristic into a principled compression engine.
Comments20 pages, 20 figures
