理解使用近似损失进行采样带来的训练加速
Understanding the Training Speedup from Sampling with Approximate Losses
AI总结：
本文提出 SIFT 方法，利用提前退出获取近似损失以降低样本选择开销，并证明贪心选择大近似损失样本在理论上可加速收敛，在 BERT 训练中显著减少训练时间。
摘要：
众所周知，选择具有较大损失或梯度的样本可以显著减少训练步数。然而，选择开销往往过高，难以在整体训练时间上产生有意义的收益。本文聚焦于一种贪心方法，即选择具有较大近似损失的样本而非精确损失，以降低选择开销。对于光滑凸损失，我们证明这种贪心策略能够在比随机选择这一标准方法更少的迭代次数内收敛到平均损失最小值的常数因子范围内。我们还从理论上量化了近似程度的影响。随后，我们开发了 SIFT，该方法利用提前退出来通过中间层的表示获得用于样本选择的近似损失。我们在训练一个具有 1.1 亿参数、12 层的 BERT base 模型的任务上评估了 SIFT，结果表明，在没有任何优化实现的情况下，相比普通训练，它在训练小时数和反向传播步数方面都取得了显著增益。例如，要达到 64% 的验证准确率，在第一层退出的 SIFT 大约需要 43 小时，而普通训练大约需要 57 小时。
英文摘要：
It is well known that selecting samples with large losses/gradients can significantly reduce the number of training steps. However, the selection overhead is often too high to yield any meaningful gains in terms of overall training time. In this work, we focus on the greedy approach of selecting samples with large \textit{approximate losses} instead of exact losses in order to reduce the selection overhead. For smooth convex losses, we show that such a greedy strategy can converge to a constant factor of the minimum value of the average loss in fewer iterations than the standard approach of random selection. We also theoretically quantify the effect of the approximation level. We then develop SIFT which uses early exiting to obtain approximate losses with an intermediate layer's representations for sample selection. We evaluate SIFT on the task of training a 110M parameter 12 layer BERT base model, and show significant gains (in terms of training hours and number of backpropagation steps) without any optimized implementation over vanilla training. For e.g., to reach 64% validation accuracy, SIFT with exit at the first layer takes ~ 43 hours compared to ~ 57 hours of vanilla training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
