mHC如何利用其残差流？选择性路由与近恒等混合
How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
- Huawei Technologies Co., Ltd.（华为技术有限公司）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
该研究探究DeepSeek-V4-Flash的四流mHC中残差流的利用模式，发现读写路由集中、残差混合主要在早期层，后期混合作用小，模型仅利用了部分灵活性。
AI中文摘要：
超连接及其流形约束变体mHC拓宽了从一条流到n条流的残差通路，但训练好的模型如何利用这一能力仍不明确：各模块读写范围有多广、残差通路混合流的强度如何、各流是否承载不同表示。我们使用有效流数、跨流残差权重和流间余弦相似度，在DeepSeek-V4-Flash的四流残差通路中研究这些属性。读写路由集中但随深度变化：典型的注意力或FFN位点有效使用约2条流，主导流随层变化，且表示保持方向上的 distinct。残差混合适度且主要发生在早期层；在22-42层，通路大多单独传递每条流。针对性干预确立了这些模式的功能重要性：用恒等替换后期混合器仅使C4困惑度增加1.9%，并保留6项任务的平均得分；而替换早期混合器使困惑度增加41%。将每个早期混合器固定到其C4诊断均值仅使困惑度增加0.2%，并使平均得分降低0.25个百分点，表明其位点特定结构比评估指标上的令牌级变化更重要。同样，在每个位点保留每个令牌的三个最大路由权重，使困惑度最多增加2.7%，平均得分最多变化0.4个点。因此，所研究模型仅实现了四流mHC所提供灵活性的一部分：单个模块很少需要所有四条流，且后期残差混合几乎无测量效益。
英文摘要：
Hyper-Connections and their manifold-constrained variant mHC widen a residual pathway from one stream to n, yet how trained models use this capacity remains unclear: how broadly blocks read and write, how strongly the residual pathway mixes streams, and whether the streams carry distinct representations. We examine these properties in the four-stream residual pathway of DeepSeek-V4-Flash using effective stream counts, cross-stream residual weights, and inter-stream cosine similarity. Read/write routing is concentrated but varies across depth: a typical attention or FFN site effectively uses about two streams, while the dominant stream changes across layers and the representations remain directionally distinct. Residual mixing is modest and occurs primarily in early layers; in layers 22-42, the pathway mostly carries each stream forward separately. Targeted interventions establish the functional significance of these patterns. Replacing the late mixers by identity increases C4 perplexity by only 1.9% and preserves the six-task average score, whereas replacing the early mixers increases perplexity by 41%. Fixing each early mixer to its C4 diagnostic mean increases perplexity by only 0.2% and reduces the average score by 0.25 percentage points, showing that its site-specific structure matters more than its token-wise variation on the evaluated metrics. Likewise, retaining the three largest routing weights per token at every site increases perplexity by at most 2.7% and changes the average score by at most 0.4 points. Thus, the studied model realizes only part of the flexibility afforded by four-stream mHC: individual blocks rarely require all four streams, and late residual mixing provides little measured benefit.
