无参数遗漏：灵敏度指导的自适应学习率用于训练大型Transformer模型
No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models
摘要：
最近的研究表明，在大型Transformer模型中存在显著的冗余。可以修剪冗余参数，而不会显着牺牲泛化性能。然而，我们质疑，如果这些冗余参数经过适当的训练，它们是否可以做出更大的贡献。为了回答这个问题，我们提出了一种新的训练策略，鼓励所有参数进行充分的训练。具体来说，我们自适应地调整学习率为每个参数根据其灵敏度，一个强大的基于梯度的措施，反映了这个参数的模型性能的贡献。敏感度低的参数是多余的，我们通过提高它的学习率来改进它的拟合。相比之下，具有高灵敏度的参数是经过良好训练的，我们通过降低其学习率来正则化它，以防止进一步的过拟合。我们在自然语言理解、神经机器翻译和图像分类上进行了大量的实验，以证明所提出的时间表的有效性。分析表明，该调度确实减少了冗余，提高了泛化性能。
英文摘要：
Recent research has shown the existence of significant redundancy in large Transformer models. One can prune the redundant parameters without significantly sacrificing the generalization performance. However, we question whether the redundant parameters could have contributed more if they were properly trained. To answer this question, we propose a novel training strategy that encourages all parameters to be trained sufficiently. Specifically, we adaptively adjust the learning rate for each parameter according to its sensitivity, a robust gradient-based measure reflecting this parameter's contribution to the model performance. A parameter with low sensitivity is redundant, and we improve its fitting by increasing its learning rate. In contrast, a parameter with high sensitivity is well-trained, and we regularize it by decreasing its learning rate to prevent further overfitting. We conduct extensive experiments on natural language understanding, neural machine translation, and image classification to demonstrate the effectiveness of the proposed schedule. Analysis shows that the proposed schedule indeed reduces the redundancy and improves generalization performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
