arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.14320cs.LG
收藏

如何训练多出口模型?分析训练策略的影响

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies

Jagiellonian University(雅盖隆大学);University of Warsaw(华沙大学);Warsaw University of Technology(华沙技术大学);IDEAS Research Institute(IDEAS研究所);Department of Information Engineering, Electronics, and Telecommunications (DIET) at Sapienza, University of Rome, Italy(罗马大学萨皮恩扎信息工程、电子与电信系);Wroclaw University of Science and Technology(波兹南科学与技术大学)

Piotr Kubaty, Bartosz Wójcik, Bartłomiej Krzepkowski, Monika Michaluk, Tomasz Trzciński, Jary Pomponi, Kamil Adamczewski

AI总结:

本文分析早期退出模型训练策略的影响,提出先训练骨干再训练整个多出口网络的混合策略,并证明其在多种架构和数据集上优于常规联合与分离训练,提升性能与效率。

摘要:

早期退出机制通过在骨干网络上附加可训练的内部分类器,使网络的前向传播能够提前终止。现有的早期退出方法通常采用联合训练方式,即骨干网络和出口头同时训练,或采用分离训练方式,即出口头单独训练。然而,这一选择的影响往往被忽视,研究通常采用某一种方式而缺乏充分论证。这一选择会影响训练动态,其影响在很大程度上尚未被探索。在本文中,我们引入一组指标来分析早期退出训练动态,并指导训练策略的选择。我们证明,常规使用的联合和分离训练机制会产生次优性能。为解决这些局限性,我们提出一种混合训练策略:首先训练骨干网络,然后训练整个多出口网络。通过对不同架构、数据集和早期退出方法下的训练策略进行全面评估,我们展示了早期退出训练策略的优缺点。特别是,我们展示了使用所提出的混合策略在性能和效率上的一致改进。

英文摘要:

Early exits enable the network's forward pass to terminate early by attaching trainable internal classifiers to the backbone network. Existing early-exit methods typically adopt either a joint training approach, where the backbone and exit heads are trained simultaneously, or a disjoint approach, where the heads are trained separately. However, the implications of this choice are often overlooked, with studies typically adopting one approach without adequate justification. This choice influences training dynamics and its impact remains largely unexplored. In this paper, we introduce a set of metrics to analyze early-exit training dynamics and guide the choice of training strategy. We demonstrate that conventionally used joint and disjoint regimes yield suboptimal performance. To address these limitations, we propose a mixed training strategy: the backbone is trained first, followed by the training of the entire multi-exit network. Through comprehensive evaluations of training strategies across various architectures, datasets, and early-exit methods, we present the strengths and weaknesses of the early exit training strategies. In particular, we show consistent improvements in performance and efficiency using the proposed mixed strategy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑