arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.01239cs.CV
收藏

Replacement Learning:用更少的可学习参数训练视觉任务

Replacement Learning: Training Vision Tasks with Fewer Learnable Parameters

Southeast University(东南大学)

Yuming Zhang, Peizhe Wang, Shouxin Zhang, Dongzhi Guan, Jiabin Liu, Junhao Su

AI总结:

本文提出 Replacement Learning,通过两个可学习参数控制相邻层参数集成并替代冻结层参数,在多个视觉基准和架构上降低参数量、训练时间与显存占用,且性能超过端到端训练。

摘要:

传统的端到端深度学习模型在训练过程中往往通过增加网络深度和复杂性来增强特征表示及整体性能。然而,这种方法不可避免地带来参数冗余和资源低效问题,尤其是在更深的网络中。尽管现有工作尝试跳过某些冗余层以缓解这些问题,但性能较差、计算复杂以及内存使用低效等挑战仍然存在。为解决这些问题,我们提出了一种名为 Replacement Learning 的创新训练方法,该方法通过仅用两个可学习参数完全替换冻结层的所有参数来缓解上述限制。具体而言,Replacement Learning 选择性地冻结某些层的参数,冻结层利用来自相邻层的参数,并通过由两个可学习参数控制的参数集成机制对其进行更新。该方法利用来自周围结构的信息,减少计算量,节省 GPU 显存,并在历史上下文与新输入之间保持平衡,最终提升模型的整体性能。我们在 CIFAR-10、STL-10、SVHN 和 ImageNet 四个基准数据集上进行了实验,使用 CNN 和 ViT 等多种架构来验证 Replacement Learning 的有效性。实验结果表明,我们的方法减少了参数数量、训练时间和内存消耗,同时性能完全超过端到端训练。

英文摘要:

Traditional end-to-end deep learning models often enhance feature representation and overall performance by increasing the depth and complexity of the network during training. However, this approach inevitably introduces issues of parameter redundancy and resource inefficiency, especially in deeper networks. While existing works attempt to skip certain redundant layers to alleviate these problems, challenges related to poor performance, computational complexity, and inefficient memory usage remain. To address these issues, we propose an innovative training approach called Replacement Learning, which mitigates these limitations by completely replacing all the parameters of the frozen layers with only two learnable parameters. Specifically, Replacement Learning selectively freezes the parameters of certain layers, and the frozen layers utilize parameters from adjacent layers, updating them through a parameter integration mechanism controlled by two learnable parameters. This method leverages information from surrounding structures, reduces computation, conserves GPU memory, and maintains a balance between historical context and new inputs, ultimately enhancing overall model performance. We conducted experiments across four benchmark datasets, including CIFAR-10, STL-10, SVHN, and ImageNet, utilizing various architectures such as CNNs and ViTs to validate the effectiveness of Replacement Learning. Experimental results demonstrate that our approach reduces the number of parameters, training time, and memory consumption while completely surpassing the performance of end-to-end training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑