HyperDFlash: 面向MHC架构的块级推测解码与门控残差缩减
HyperDFlash: Hyper-Connection-Aligned Block Speculative Decoding with Gated Residual Reduction
AI总结：
针对DeepSeek-V4的多超连接架构，提出HyperDFlash框架，通过对齐残差流和轻量门控缩减器，解决推测解码中特征错位和误差累积问题，显著提升解码速度。
摘要：
我们提出了HyperDFlash，这是一个面向DeepSeek-V4提出的新型多超连接（MHC）架构的块并行推测解码框架。尽管DeepSeek-V4原生多令牌预测（MTP）模块具有强大的初始令牌草稿生成能力，但其草稿准确率在后续位置急剧下降，因为未验证中间令牌的误差累积损害了接受率。虽然原始的DFlash方法支持高效的单遍块草稿生成，但它无法无缝适配MHC范式，因为DeepSeek-V4的多路径残差流导致与常规草稿设计产生特征错位。为解决这一不匹配问题，我们针对MHC残差流提出了两种模型对齐的优化。首先，我们采用预折叠残差状态作为唯一的条件信号，保留多路径结构信息，使草稿模型与目标模型的原生预测路径对齐。其次，我们用轻量级门控残差缩减器替代笨重的通用线性压缩器，其参数继承自内置的超连接头。该设计实现了输入感知的路径聚合，参数数量减少三个数量级，同时保持架构对齐。我们进一步通过针对LM头的定向KL蒸馏损失来增强训练，该损失针对完整目标概率分布正则化预测，并在早期训练阶段提高草稿质量。在数学推理、代码合成和对话基准上的实验表明，HyperDFlash始终优于原生MTP基线和普通DFlash适配。它在平均接受草稿长度和解码加速方面取得了显著提升，验证了MHC对齐、门控缩减和定向蒸馏在高性能推测解码中的有效性。
英文摘要：
We present HyperDFlash, a block-parallel speculative decoding framework tailored to DeepSeek-V4's Hyper-Connections (HC). Despite the strong performance of DeepSeek-V4's native Multi-Token Prediction (MTP) module on initial token drafting, its draft accuracy degrades sharply at later positions, as error accumulation from unverified intermediate tokens harms draft acceptance rates. Although the original DFlash method supports efficient one-pass block drafting, it cannot be seamlessly adapted to the HC paradigm, since DeepSeek-V4's multi-path residual stream induces inherent feature misalignment with conventional drafting designs. To resolve this architectural mismatch, we propose two dedicated, model-aligned optimizations for HC residual streams. First, we adopt pre-collapse residual states as the exclusive conditioning signal, preserving complete multi-path structural information and better aligning the drafter with the target's native prediction pathway. Second, we replace the heavy generic linear compressor with a lightweight gated residual reducer, whose parameters are directly inherited from the target model's built-in hc_head module. This design yields input-aware path aggregation with three orders of magnitude fewer parameters while maintaining precise architectural alignment. We further enhance model training via a targeted KL distillation loss applied to the LM-head, regularizing predictions against the target distribution to improve early draft quality. Extensive experiments across math reasoning, code synthesis, and conversational benchmarks demonstrate that HyperDFlash consistently outperforms both the native MTP baseline and vanilla DFlash adaptation, achieving substantial gains in average accepted draft length and decoding speedup. These results validate HC alignment, gated reduction, and targeted distillation for high-performance speculative decoding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
