分析超连接中的流坍缩：从诊断到缓解
Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation
AI总结：
本文通过细粒度诊断发现超连接中的多流残差连接存在流坍缩现象，即信号集中于主导流，并通过打破初始化对称性缓解该问题以提升性能。
摘要：
超连接（HC）用多个流替换单个Transformer残差流，引入了流索引上的置换对称性。我们研究这种对称性在实践中如何被打破：流是平衡地专门化还是表现出主导流使用。通过对基于HC的语言模型进行细粒度诊断，我们追踪多流表示的实际使用方式。我们发现，在早期种子阶段之后，残差混合通常保持接近恒等映射，限制了HC在流之间交换信息的核心机制。此外，信号和可解释特征都集中在一个主导流中，名义上的多流残差连接可能未充分利用其容量，行为更接近单流残差路径。最后，我们表明在流初始化时打破对称性可以减少主导行为并提高各种 extit{m}HC变体的性能。我们的代码已公开。
英文摘要：
Hyper-Connections (HC) replace the single Transformer residual stream with multiple streams, introducing a permutation symmetry over stream indices. We study how this symmetry is resolved in practice: whether streams specialize in a balanced way or exhibit dominant-stream usage. Using fine-grained diagnostics for HC-based language models, we trace how multi-stream representations are actually used. We find that after an early seeding stage, residual mixing often remains close to identity, limiting a core HC mechanism for exchanging information between streams. Moreover, both signal and interpretable features concentrate in a dominant stream, and the nominally multi-stream residual connection can underutilize its capacity, behaving closer to a single-stream residual pathway. Finally, we show that breaking symmetry at stream initialization reduces dominant behavior and improves performance across \textit{m}HC variants. Our code is publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
