arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.13337cs.LGcs.AIstat.ML
收藏

揭秘秘方:小型 LLMs 监督微调指南

Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs

Red Hat AI Innovation(红帽人工智能创新中心);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);IBM Research(IBM研究院);Massachusetts Institute of Technology(麻省理工学院);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校)
Aldo Pareja, Nikhil Shivakumar Nayak, Hao Wang, Krishnateja Killamsetty, Shivchander Sudalairaj, Wenlong Zhao, Seungwook Han, Abhishek Bhandwaldar, Guangxuan Xu… 展开作者

Aldo Pareja, Nikhil Shivakumar Nayak, Hao Wang, Krishnateja Killamsetty, Shivchander Sudalairaj, Wenlong Zhao, Seungwook Han, Abhishek Bhandwaldar, Guangxuan Xu, Kai Xu, Ligong Han, Luke Inglis, Akash Srivastava

AI总结:

本文针对资源受限的小型组织,研究了3B至7B参数小型LLMs的监督微调,发现大批次配低学习率、早期训练动态等可优化性能并节省计算,挑战了既有训练实践。

摘要:

大型语言模型(LLMs)的崛起造成了显著差距:拥有计算资源、专家团队和先进基础设施的工业研究实验室能够有效微调 LLMs,而个人开发者和小型组织则因资源有限面临障碍。本文旨在通过一项关于使用跨越不同知识领域和技能的指令微调数据集对 LLMs 进行监督微调的全面研究来弥合这一差距。我们关注参数规模在 3B 到 7B 之间的小型 LLMs,因为它们具有成本效益和可及性。我们在四个开源预训练模型上探索了各种训练配置和策略。我们提供了这些配置的详细文档,揭示了挑战几种常见训练实践的发现,包括 TULU 的超参数建议和 Orca 推荐的分阶段训练。我们工作的关键见解包括:(i) 更大的批次大小配合更低的学习率,能在 MMLU、MTBench 和 Open LLM Leaderboard 等基准上带来更好的模型性能;(ii) 早期训练动态,如较低的梯度范数和较高的损失值,是最终模型性能更佳的强有力指标,能够提前终止次优运行并显著节省计算量;(iii) 通过对预热步数和学习率调度等超参数的深入探索,我们为从业者提供了指导,并发现某些简化不会损害性能;(iv) 我们观察到分阶段和堆叠训练策略之间没有显著的性能差异,但堆叠训练更简单且样本效率更高。这些发现在不同数据集和模型上均稳健成立,我们希望本研究能作为从业者微调小型 LLMs 的指南,并促进 LLM 研究更具包容性的环境。

英文摘要:

The rise of large language models (LLMs) has created a significant disparity: industrial research labs with their computational resources, expert teams, and advanced infrastructures, can effectively fine-tune LLMs, while individual developers and small organizations face barriers due to limited resources. In this paper, we aim to bridge this gap by presenting a comprehensive study on supervised fine-tuning of LLMs using instruction-tuning datasets spanning diverse knowledge domains and skills. We focus on small-sized LLMs (3B to 7B parameters) for their cost-efficiency and accessibility. We explore various training configurations and strategies across four open-source pre-trained models. We provide detailed documentation of these configurations, revealing findings that challenge several common training practices, including hyperparameter recommendations from TULU and phased training recommended by Orca. Key insights from our work include: (i) larger batch sizes paired with lower learning rates lead to improved model performance on benchmarks such as MMLU, MTBench, and Open LLM Leaderboard; (ii) early-stage training dynamics, such as lower gradient norms and higher loss values, are strong indicators of better final model performance, enabling early termination of sub-optimal runs and significant computational savings; (iii) through a thorough exploration of hyperparameters like warmup steps and learning rate schedules, we provide guidance for practitioners and find that certain simplifications do not compromise performance; and (iv) we observed no significant difference in performance between phased and stacked training strategies, but stacked training is simpler and more sample efficient. With these findings holding robustly across datasets and models, we hope this study serves as a guide for practitioners fine-tuning small LLMs and promotes a more inclusive environment for LLM research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑