数据科学与技术迈向通用人工智能 Part I：分层数据管理
Data Science and Technology Towards AGI Part I: Tiered Data Management
AI总结：
本文提出分层数据管理框架，通过数据与模型的协同进化提升大语言模型训练效率和性能。
摘要：
人工智能的发展可以被视为数据驱动学习范式的演变，连续的数据组织和利用的转变不断推动模型能力的进步。当前的大语言模型研究主要由一种依赖数据规模单向扩展的范式主导，日益遇到数据可用性、获取成本和训练效率的瓶颈。在本工作中，我们认为通用人工智能的发展进入了一个新的数据-模型共进化阶段，其中模型主动引导数据管理，而高质量的数据反过来增强模型能力。为实现这一愿景，我们提出了一种分层数据管理框架，旨在支持跨异构学习目标和成本约束的完整大语言模型训练生命周期。具体来说，我们引入了一个L0-L4分层数据管理框架，从原始未经整理的资源到组织和可验证的知识。重要的是，大语言模型被完全用于数据管理过程，如质量评分和内容编辑，以在各层级上优化数据。每个层级具有不同的数据属性、管理策略和训练角色，使数据能够战略地分配到大语言模型训练阶段，包括预训练、中训练和对齐。该框架平衡了数据质量、获取成本和边际训练收益，提供了一种可扩展和可持续的数据管理系统方法。我们通过实证研究验证了所提框架的有效性，在其中构建了分层数据集并用于多个训练阶段。实验结果表明，分层数据利用显著提高了训练效率和模型性能。为了促进进一步研究，我们向社区发布了我们的分层数据集和处理工具。
英文摘要：
The development of artificial intelligence can be viewed as an evolution of data-driven learning paradigms, with successive shifts in data organization and utilization continuously driving advances in model capability. Current LLM research is dominated by a paradigm that relies heavily on unidirectional scaling of data size, increasingly encountering bottlenecks in data availability, acquisition cost, and training efficiency. In this work, we argue that the development of AGI is entering a new phase of data-model co-evolution, in which models actively guide data management while high-quality data, in turn, amplifies model capabilities. To implement this vision, we propose a tiered data management framework, designed to support the full LLM training lifecycle across heterogeneous learning objectives and cost constraints. Specifically, we introduce an L0-L4 tiered data management framework, ranging from raw uncurated resources to organized and verifiable knowledge. Importantly, LLMs are fully used in data management processes, such as quality scoring and content editing, to refine data across tiers. Each tier is characterized by distinct data properties, management strategies, and training roles, enabling data to be strategically allocated across LLM training stages, including pre-training, mid-training, and alignment. The framework balances data quality, acquisition cost, and marginal training benefit, providing a systematic approach to scalable and sustainable data management. We validate the effectiveness of the proposed framework through empirical studies, in which tiered datasets are constructed from raw corpora and used across multiple training phases. Experimental results demonstrate that tier-aware data utilization significantly improves training efficiency and model performance. To facilitate further research, we release our tiered datasets and processing tools to the community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
