arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.01700cs.CLcs.AI
收藏

训练大语言模型的数据管理:综述

Data Management For Training Large Language Models: A Survey

School of Computer Science, Peking University(北京大学计算机学院)

Zige Wang, Wanjun Zhong, Yufei Wang, Qi Zhu, Fei Mi, Baojun Wang, Lifeng Shang, Xin Jiang, Qun Liu

AI总结:

本文综述了大语言模型预训练与监督微调阶段的数据管理研究,系统梳理策略设计、现有挑战与发展方向,为高效构建强大LLM提供指导。

摘要:

数据在训练大语言模型(LLMs)中发挥着基础性作用。高效的数据管理,特别是构建合适的数据集,对于在预训练和监督微调阶段提升模型性能与训练效率具有重要意义。尽管数据管理十分重要,当前主流实践背后的机制仍不明确。因此,数据管理的探索日益受到研究界的关注。本综述旨在全面梳理LLMs预训练和监督微调阶段数据管理的研究现状,涵盖数据管理策略设计的各个方面。展望未来,我们推断了现有挑战并勾勒出该领域有前景的发展方向。因此,本综述可作为希望通过高效数据管理实践构建强大LLMs的从业者的指导资源。最新论文集合可在 https://github.com/ZigeW/data_management_LLM 获取。

英文摘要:

Data plays a fundamental role in training Large Language Models (LLMs). Efficient data management, particularly in formulating a well-suited training dataset, is significant for enhancing model performance and improving training efficiency during pretraining and supervised fine-tuning stages. Despite the considerable importance of data management, the underlying mechanism of current prominent practices are still unknown. Consequently, the exploration of data management has attracted more and more attention among the research community. This survey aims to provide a comprehensive overview of current research in data management within both the pretraining and supervised fine-tuning stages of LLMs, covering various aspects of data management strategy design. Looking into the future, we extrapolate existing challenges and outline promising directions for development in this field. Therefore, this survey serves as a guiding resource for practitioners aspiring to construct powerful LLMs through efficient data management practices. The collection of the latest papers is available at https://github.com/ZigeW/data_management_LLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑