arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.16737cs.CLcs.AI
收藏

更小、更弱却更优:通过计算最优采样训练大语言模型推理器

Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling

Google DeepMind(谷歌DeepMind);UCLA(加利福尼亚大学洛杉矶分校);Mila(米拉计算研究所)

Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q. Tran, Mehran Kazemi

AI总结:

该研究针对固定推理预算下大语言模型推理训练的计算最优性问题,对比强成本高与弱成本低模型生成的合成数据,发现弱成本低模型数据微调的模型性能更优,挑战了现有主流做法。

摘要:

利用强语言模型(LM)生成的高质量合成数据进行训练,是提升大语言模型推理性能的常用策略。本研究重新探讨了在固定推理预算(如FLOPs)下,该策略是否具备计算最优性。为此,我们研究了使用更强但成本更高(SE)的模型与更弱但成本更低(WC)的模型生成合成数据之间的权衡关系。我们从覆盖率、多样性和假阳性率三个关键指标评估生成的数据,发现WC模型生成的数据可能具有更高的覆盖率和多样性,但假阳性率也更高。随后,我们在三种不同设置下,分别用SE和WC模型生成的数据微调语言模型:知识蒸馏、自我改进,以及一种新颖的弱到强改进设置——由更弱的LM向更强的LM传授推理能力。研究结果表明,在多个基准测试以及多种WC和SE模型的选择组合中,用WC生成数据微调的模型始终优于用SE生成数据训练的模型。这些结果对依赖SE模型生成合成数据的主流做法提出了挑战,表明WC可能是训练高级大语言模型推理器的计算最优方案。

英文摘要:

Training on high-quality synthetic data from strong language models (LMs) is a common strategy to improve the reasoning performance of LMs. In this work, we revisit whether this strategy is compute-optimal under a fixed inference budget (e.g., FLOPs). To do so, we investigate the trade-offs between generating synthetic data using a stronger but more expensive (SE) model versus a weaker but cheaper (WC) model. We evaluate the generated data across three key metrics: coverage, diversity, and false positive rate, and show that the data from WC models may have higher coverage and diversity, but also exhibit higher false positive rates. We then finetune LMs on data from SE and WC models in different settings: knowledge distillation, self-improvement, and a novel weak-to-strong improvement setup where a weaker LM teaches reasoning to a stronger LM. Our findings reveal that models finetuned on WC-generated data consistently outperform those trained on SE-generated data across multiple benchmarks and multiple choices of WC and SE models. These results challenge the prevailing practice of relying on SE models for synthetic data generation, suggesting that WC may be the compute-optimal approach for training advanced LM reasoners.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑