arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.12178cs.LGstat.ML
收藏

模型平衡有助于低数据训练和微调

Model Balancing Helps Low-data Training and Fine-tuning

Dartmouth College(达特茅斯学院);University of California, Berkeley(加州大学伯克利分校);University of California, San Diego(加州大学圣地亚哥分校);Lawrence Berkeley National Lab(劳伦斯伯克利国家实验室)

Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, Yaoqing Yang

AI总结:

针对低数据训练和微调中不同模型层训练质量不平衡的问题,基于HT-SR理论提出分层学习率调度器TempBalance,有效平衡各层训练质量,在NLP和SciML任务中显著提升性能,且数据越少提升越大。

摘要:

基础模型的最新进展强调了使用小型精选数据集将预训练模型与特定领域对齐的需求。对这些基础模型的研究突显了低数据训练和微调的重要性。这一在自然语言处理(NLP)中广为人知的话题,在新兴的科学机器学习(SciML)领域也获得了越来越多的关注。为了解决低数据训练和微调的局限性,我们从重尾自正则化(HT-SR)理论中汲取灵感,分析了经验谱密度(ESD)的形状,揭示了不同模型层之间训练质量的不平衡。为了缓解这一问题,我们采用了最近提出的分层学习率调度器 TempBalance,它有效地平衡了各层的训练质量,并增强了 NLP 和 SciML 任务的低数据训练和微调。值得注意的是,随着可用微调数据量的减少,TempBalance 展现出越来越大的性能提升。比较分析进一步突显了 TempBalance 的有效性及其作为提升模型性能的“附加”方法的适应性。

英文摘要:

Recent advances in foundation models have emphasized the need to align pre-trained models with specialized domains using small, curated datasets. Studies on these foundation models underscore the importance of low-data training and fine-tuning. This topic, well-known in natural language processing (NLP), has also gained increasing attention in the emerging field of scientific machine learning (SciML). To address the limitations of low-data training and fine-tuning, we draw inspiration from Heavy-Tailed Self-Regularization (HT-SR) theory, analyzing the shape of empirical spectral densities (ESDs) and revealing an imbalance in training quality across different model layers. To mitigate this issue, we adapt a recently proposed layer-wise learning rate scheduler, TempBalance, which effectively balances training quality across layers and enhances low-data training and fine-tuning for both NLP and SciML tasks. Notably, TempBalance demonstrates increasing performance gains as the amount of available tuning data decreases. Comparative analyses further highlight the effectiveness of TempBalance and its adaptability as an "add-on" method for improving model performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑