自动压缩网络
Auto-Compressing Networks
AI总结：
本文提出自动压缩网络（ACNs），用逐层到输出的长前馈连接替代短残差连接，使网络在训练中自动压缩信息、动态利用各层，从而在保持准确率的同时实现30–80%架构压缩，并提升噪声鲁棒性、迁移学习能力及缓解灾难性遗忘。
摘要：
具有短残差连接的深度神经网络已在多个领域展现出显著成功，但不断增加深度往往引入计算冗余，而表征质量并未相应提升。我们提出自动压缩网络（ACNs），这是一种架构变体，其中从每一层到输出的加性长前馈连接取代了传统的短残差连接。通过分析这一修改所诱导的不同动力学，我们揭示了一种独特性质，并将其命名为自动压缩，即网络仅通过架构设计就能够在梯度下降训练过程中有机地压缩信息。借助自动压缩，信息在训练期间被动态地“推入”早期层，从而提升其表征质量，并揭示更深层中可能存在的冗余。我们从理论上证明，该性质源于ACNs中存在的逐层训练模式，其中各层在训练期间根据任务需求被动态利用。我们还发现，与残差网络相比，ACNs表现出更强的噪声鲁棒性、在低数据场景下更优的性能、改进的迁移学习能力，并能缓解灾难性遗忘，这表明它们在使用更少参数的情况下学习到了泛化能力更好的表征。我们的结果表明，在视觉Transformer、MLP-Mixer和BERT架构上，灾难性遗忘最多降低18%，架构压缩率达到30–80%，同时保持准确率。这些发现使ACNs成为一种实用方法，可用于开发高效神经架构，使其计算足迹自动适应任务复杂度，同时学习适用于噪声真实世界任务和持续学习场景的鲁棒表征。
英文摘要：
Deep neural networks with short residual connections have demonstrated remarkable success across domains, but increasing depth often introduces computational redundancy without corresponding improvements in representation quality. We introduce Auto-Compressing Networks (ACNs), an architectural variant where additive long feedforward connections from each layer to the output replace traditional short residual connections. By analyzing the distinct dynamics induced by this modification, we reveal a unique property we coin as auto-compression, the ability of a network to organically compress information during training with gradient descent, through architectural design alone. Through auto-compression, information is dynamically "pushed" into early layers during training, enhancing their representational quality and revealing potential redundancy in deeper ones. We theoretically show that this property emerges from layer-wise training patterns present in ACNs, where layers are dynamically utilized during training based on task requirements. We also find that ACNs exhibit enhanced noise robustness compared to residual networks, superior performance in low-data settings, improved transfer learning capabilities, and mitigate catastrophic forgetting suggesting that they learn representations that generalize better despite using fewer parameters. Our results demonstrate up to 18% reduction in catastrophic forgetting and 30-80% architectural compression while maintaining accuracy across vision transformers, MLP-mixers, and BERT architectures. These findings establish ACNs as a practical approach to developing efficient neural architectures that automatically adapt their computational footprint to task complexity, while learning robust representations suitable for noisy real-world tasks and continual learning scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
