arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.09475cs.LGcs.AI
收藏

ResidualDroppath:增强残差连接上的特征复用

ResidualDroppath: Enhancing Feature Reuse over Residual Connections

Graduate School of AI, Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院人工智能研究生院)

Sejik Park

AI总结:

针对原始残差连接在特征复用上的局限性,提出ResidualDroppath训练方法,通过droppath随机丢弃层并交替训练丢弃与冻结部分以增强特征复用,在特定图像分类任务中提升了模型性能。

摘要:

残差连接是神经网络架构中最重要的组件之一,用于缓解梯度消失问题并促进更深网络的训练。关于残差连接如何辅助更深网络训练的一种可能解释是促进特征复用。然而,我们识别并分析了原始残差连接在特征复用方面的局限性。为了解决这些局限性,我们提出了训练方法的修改。具体而言,我们通过训练期间的两种迭代,为模型提供了通过残差连接学习特征复用的额外机会。第一类迭代涉及使用droppath,它通过随机丢弃一部分层来强制特征复用。第二类迭代侧重于训练模型中被丢弃的部分,同时冻结未被丢弃的部分。因此,被丢弃的部分以鼓励特征复用的方式进行学习,因为模型依赖于未被丢弃的部分并考虑到特征复用。总体而言,我们证明了在某些情况下,具有残差连接的模型在图像分类任务中取得了性能提升。

英文摘要:

Residual connections are one of the most important components in neural network architectures for mitigating the vanishing gradient problem and facilitating the training of much deeper networks. One possible explanation for how residual connections aid deeper network training is by promoting feature reuse. However, we identify and analyze the limitations of feature reuse with vanilla residual connections. To address these limitations, we propose modifications in training methods. Specifically, we provide an additional opportunity for the model to learn feature reuse with residual connections through two types of iterations during training. The first type of iteration involves using droppath, which enforces feature reuse by randomly dropping a subset of layers. The second type of iteration focuses on training the dropped parts of the model while freezing the undropped parts. As a result, the dropped parts learn in a way that encourages feature reuse, as the model relies on the undropped parts with feature reuse in mind. Overall, we demonstrated performance improvements in models with residual connections for image classification in certain cases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑