arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.26164cs.LGcs.CL
收藏

DataFlex: 一种统一的以数据为中心的大型语言模型动态训练框架

DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models

Peking University（北京大学）；Institute for Advanced Algorithms Research, Shanghai（上海先进算法研究所）；OriginHub Technology；LLaMA-Factory Team（LLaMA-Factory团队）；Zhongguancun Academy（中关村学院）；OpenDataLab, Shanghai Artificial Intelligence Laboratory（上海人工智能实验室OpenDataLab）；Shanghai Jiao Tong University（上海交通大学）
Hao Liang, Zhengyang Zhao, Mingrui Chen, Meiyi Qiang, Lu Ma, Rongyi Yu, Hengyi Feng, Shixuan Sun, Zimo Meng, Xiaochen Ma, Xuanlin Yang, Qifeng Cai, Ruichuan An,… 展开作者

Hao Liang, Zhengyang Zhao, Mingrui Chen, Meiyi Qiang, Lu Ma, Rongyi Yu, Hengyi Feng, Shixuan Sun, Zimo Meng, Xiaochen Ma, Xuanlin Yang, Qifeng Cai, Ruichuan An, Bohan Zeng, Zhen Hao Wong, Chengyu Shen, Runming He, Zhaoyang Han, Yaowei Zheng, Fangcheng Fu, Conghui He, Bin Cui, Zhiyu Li, Weinan E, Wentao Zhang

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

DataFlex 提供了一种统一的以数据为中心的动态训练框架，支持样本选择、领域混合调整和样本再加权，提升了大型语言模型的训练效率和可重复性。

摘要：

以数据为中心的训练已成为改进大型语言模型（LLMs）的有前景方向，通过优化不仅模型参数，还包括训练数据的选择、组成和加权。然而，现有的数据选择、数据混合优化和数据再加权方法往往在孤立的代码库中开发，接口不一致，阻碍了可重复性、公平比较和实际集成。本文提出了DataFlex，一种基于LLaMA-Factory构建的统一以数据为中心的动态训练框架。DataFlex支持三种主要的动态数据优化范式：样本选择、领域混合调整和样本再加权，同时保持与原始训练流程的完全兼容。它提供了可扩展的训练器抽象和模块化组件，使标准LLM训练能够无缝替换，并统一了关键的模型依赖操作，如嵌入提取、推理和梯度计算，支持大规模设置，包括DeepSpeed ZeRO-3。我们进行了多项数据为中心方法的全面实验。动态数据选择在MMLU上在Mistral-7B和Llama-3.2-3B上均优于静态全数据训练。对于数据混合，DoReMi和ODM在预训练Qwen2.5-1.5B时，通过在SlimPajama上6B和30B token规模上改进MMLU准确率和语料库级困惑度。DataFlex还实现了与原始实现相比的一致运行时间改进。这些结果表明，DataFlex为大型语言模型的数据为中心动态训练提供了有效的、高效的和可重复的基础设施。

英文摘要：

Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters but also the selection, composition, and weighting of training data during optimization. However, existing approaches to data selection, data mixture optimization, and data reweighting are often developed in isolated codebases with inconsistent interfaces, hindering reproducibility, fair comparison, and practical integration. In this paper, we present DataFlex, a unified data-centric dynamic training framework built upon LLaMA-Factory. DataFlex supports three major paradigms of dynamic data optimization: sample selection, domain mixture adjustment, and sample reweighting, while remaining fully compatible with the original training workflow. It provides extensible trainer abstractions and modular components, enabling a drop-in replacement for standard LLM training, and unifies key model-dependent operations such as embedding extraction, inference, and gradient computation, with support for large-scale settings including DeepSpeed ZeRO-3. We conduct comprehensive experiments across multiple data-centric methods. Dynamic data selection consistently outperforms static full-data training on MMLU across both Mistral-7B and Llama-3.2-3B. For data mixture, DoReMi and ODM improve both MMLU accuracy and corpus-level perplexity over default proportions when pretraining Qwen2.5-1.5B on SlimPajama at 6B and 30B token scales. DataFlex also achieves consistent runtime improvements over original implementations. These results demonstrate that DataFlex provides an effective, efficient, and reproducible infrastructure for data-centric dynamic training of LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑