消除与救援:残差流超连接的因果分析
Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections
AI总结:
本文提出首个开源mHC语言模型,通过代表层面指标和因果干预分析多流架构,揭示并行流的信息编码与利用机制,引入系统性的流消除-救援框架进行因果比较。
摘要:
多流Transformer架构最近被提出作为解决表示崩溃和残差连接消失梯度问题的有前途方向,但其内部机制仍不清楚。特别是最近引入的Manifold-Constrained Hyper-Connections(mHC)架构提出多个具有受限交互的残差流,但缺乏深入的机理分析。我们提出了首个开源mHC语言模型(https://huggingface.co/wgpeng/mhc-780m)并利用一系列代表层面指标和因果干预来分析多流架构,以探究并行流如何编码和利用信息。具体而言,我们引入了系统性的流消除-救援框架,该框架在推理过程中能够直接比较残差流的因果性。通过针对性的成对干预和受控恢复实验,我们区分了功能冗余与不对称利用,并揭示了信息如何在流之间分布,这超出了仅基于表示相似性所能观察到的范围。
英文摘要:
Multi-stream transformer architectures have recently been proposed as a promising direction for managing representation collapse and the vanishing gradient problem for residual connections, yet their internal mechanisms remain unexplored. In particular, the recently introduced Manifold-Constrained Hyper-Connections (mHC) architecture posits multiple residual streams with constrained interaction, but lacks in-depth mechanistic analysis. We present the first open-source mHC language model (https://huggingface.co/wgpeng/mhc-780m) and analyze the multiple-stream architecture with a suite of representation-level metrics and causal interventions to probe how parallel streams encode and utilize information. Specifically, we introduce a systematic stream ablation-and-rescue framework that enables direct causal comparison of residual streams during inference. Through targeted pairwise interventions and controlled recovery experiments, we distinguish functional redundancy from asymmetric utilization and reveal how information is distributed across streams beyond what is observable from representational similarity alone.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。