迈向下一代大语言模型训练:从数据导向的视角
Towards Next-Generation LLM Training: From the Data-Centric Perspective
AI总结:
本文从数据导向视角出发,探讨了大语言模型训练中的数据准备瓶颈问题,提出构建自动数据准备系统和统一的数据-模型交互训练系统,以提升数据利用效率和模型性能。
摘要:
大语言模型(LLMs)在广泛任务和领域中展示了卓越的性能,数据在其中扮演着核心角色。尽管取得成功,LLM训练所需的大量数据的准备和有效利用仍是主要瓶颈。当前实践中,LLM训练数据通常通过随意脚本构建,缺乏成熟的、基于代理的数据准备系统,无法自动构建稳健且可重用的数据流程,导致数据科学家需重复且易出错的工程工作。此外,一旦收集,数据往往在训练过程中被整体消耗,缺乏系统性的数据选择、混合优化或重新加权机制。为解决这些局限,我们倡导两个互补的研究方向。首先,我们提出构建稳健、基于代理的自动数据准备系统,支持自动化工作流程构建和可扩展的数据管理。其次,我们主张统一的数据-模型交互训练系统,在训练过程中动态选择、混合和重新加权数据,从而实现更高效、适应性和性能感知的数据利用。最后,我们讨论了剩余的挑战并概述了未来研究和系统开发的有前途的方向。
英文摘要:
Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks and domains, with data playing a central role in enabling these advances. Despite this success, the preparation and effective utilization of the massive datasets required for LLM training remain major bottlenecks. In current practice, LLM training data is often constructed using ad hoc scripts, and there is still a lack of mature, agent-based data preparation systems that can automatically construct robust and reusable data workflows, thereby freeing data scientists from repetitive and error-prone engineering efforts. Moreover, once collected, datasets are often consumed largely in their entirety during training, without systematic mechanisms for data selection, mixture optimization, or reweighting. To address these limitations, we advocate two complementary research directions. First, we propose building a robust, agent-based automatic data preparation system that supports automated workflow construction and scalable data management. Second, we argue for a unified data-model interaction training system in which data is dynamically selected, mixed, and reweighted throughout the training process, enabling more efficient, adaptive, and performance-aware data utilization. Finally, we discuss the remaining challenges and outline promising directions for future research and system development.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。