Landscape-Aware Growing：一点 LAG 的力量
Landscape-Aware Growing: The Power of a Little LAG
AI总结：
本文提出了一种基于早期训练动态的 landscape-aware growing (LAG) 方法，用于更准确地选择最佳的模型 growing 策略，并启发了一种用于 gradual stacking 的自适应策略。
摘要：
近期，针对基于 Transformer 的模型的高效预训练范式引起了越来越多的关注。一些近期的方法使用较小的模型来初始化较大的模型以节省计算量（例如，stacking 和 fusion）。在这项工作中，我们研究了一个基本问题：如何从给定的 growing 策略池中选择最佳的 growing 策略。先前的工作广泛聚焦于初始化时保持 loss 或功能不变的行为，或者仅仅是训练结束时的性能。相反，我们发现初始化时的行为作为最终性能的预测指标可能具有误导性，并提出了一种基于早期训练动态的替代视角，我们称之为“landscape-aware growing (LAG)”。我们对最终性能与训练初始步骤中性能的相关性进行了广泛分析，发现能够更早且更准确地预测最佳的 growing 策略（即，在初始化后仅有少量的“lag”）。这一视角也启发了一种用于 gradual stacking 的自适应策略。
英文摘要：
Recently, there has been increasing interest in efficient pretraining paradigms for training Transformer-based models. Several recent approaches use smaller models to initialize larger models in order to save computation (e.g., stacking and fusion). In this work, we study the fundamental question of how to select the best growing strategy from a given pool of growing strategies. Prior works have extensively focused on loss- and/or function-preserving behavior at initialization or simply performance at the end of training. Instead, we identify that behavior at initialization can be misleading as a predictor of final performance and present an alternative perspective based on early training dynamics, which we call "landscape-aware growing (LAG)". We perform extensive analysis of correlation of the final performance with performance in the initial steps of training and find early and more accurate predictions of the optimal growing strategy (i.e., with only a small "lag" after initialization). This perspective also motivates an adaptive strategy for gradual stacking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
