ANCRe: 适应性神经连接重新分配用于高效的深度扩展
ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
AI总结：
ANCRe通过适应性神经连接重新分配，有效提升深度扩展效率，加速收敛并提高性能。
摘要：
网络深度的扩展一直是现代基础模型成功的关键驱动力，但最近的研究表明，深层网络往往被低估。本文重新审视了加深神经网络的默认机制，即残差连接，从优化的角度出发。严谨的分析证明，残差连接的布局可以根本上塑造收敛行为，甚至会导致收敛速率的指数级差距。受此启发，我们引入了适应性神经连接重新分配（ANCRe），这是一种原理性强且轻量的框架，它参数化并从数据中学习残差连接性。ANCRe 以几乎不增加计算和内存开销（<1%）的方式适应性地重新分配残差连接，同时使网络深度的利用更加有效。在大规模语言模型预训练、扩散模型和深度 ResNets 的广泛数值测试中，一致地实现了加速收敛、性能提升和传统残差连接的深度效率提升。
英文摘要：
Scaling network depth has been a central driver behind the success of modern foundation models, yet recent investigations suggest that deep layers are often underutilized. This paper revisits the default mechanism for deepening neural networks, namely residual connections, from an optimization perspective. Rigorous analysis proves that the layout of residual connections can fundamentally shape convergence behavior, and even induces an exponential gap in convergence rates. Prompted by this insight, we introduce adaptive neural connection reassignment (ANCRe), a principled and lightweight framework that parameterizes and learns residual connectivities from the data. ANCRe adaptively reassigns residual connections with negligible computational and memory overhead ($<1\%$), while enabling more effective utilization of network depth. Extensive numerical tests across pre-training of large language models, diffusion models, and deep ResNets demonstrate consistently accelerated convergence, boosted performance, and enhanced depth efficiency over conventional residual connections.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
