arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.11089cs.CLcs.AI
收藏

DataChef: 通过强化学习为LLM适应生成最优数据配方

DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learning

Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室)

Yicheng Chen, Zerun Ma, Xinchen Xie, Yining Li, Kai Chen

AI总结:

DataChef通过强化学习自动生成最优数据配方,使LLM适应性能达到人类专家水平。

摘要:

在当前大型语言模型(LLM)的背景下,大规模、高质量训练数据的编纂是模型性能的主要驱动力。关键的杠杆是数据配方,它由数据处理管道组成,用于将原始数据源转换为训练语料库。尽管LLM被广泛用于自动化单个数据处理步骤,如数据合成和过滤,但数据配方的整体设计仍然很大程度上是手动和劳动密集型的,需要大量的人类专业知识和迭代。为了弥合这一差距,我们提出了端到端的数据配方生成,用于LLM适应。给定一个目标基准和一个可用的数据源池,模型需要输出一个完整的数据配方,以适应基础LLM到目标任务。我们提出了DataChef-32B,它使用代理奖励进行在线强化学习,该奖励预测候选配方的下游性能。在六个保留任务上,DataChef-32B产生的配方在性能上与由人类专家编纂的配方相当。值得注意的是,DataChef-32B生成的配方将Qwen3-1.7B-Base适应到数学领域,达到AIME'25的66.7分,并超过了官方后训练检查点(Qwen3-1.7B)。这项工作为自动化LLM训练和开发自我进化的AI系统提供了新的视角。

英文摘要:

In the current landscape of Large Language Models (LLMs), the curation of large-scale, high-quality training data is a primary driver of model performance. A key lever is the \emph{data recipe}, which comprises a data processing pipeline to transform raw sources into training corpora. Despite the growing use of LLMs to automate individual data processing steps, such as data synthesis and filtering, the overall design of data recipes remains largely manual and labor-intensive, requiring substantial human expertise and iteration. To bridge this gap, we formulate \emph{end-to-end data recipe generation} for LLM adaptation. Given a target benchmark and a pool of available data sources, a model is required to output a complete data recipe that adapts a base LLM to the target task. We present DataChef-32B, which performs online reinforcement learning using a proxy reward that predicts downstream performance for candidate recipes. Across six held-out tasks, DataChef-32B produces recipes that yield performance comparable to those curated by human experts. Notably, the recipe from DataChef-32B adapts Qwen3-1.7B-Base to the math domain, achieving 66.7 on AIME'25 and surpassing the official post-training checkpoint (Qwen3-1.7B). This work sheds new light on automating LLM training and developing self-evolving AI systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑