arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.00997cs.CL
收藏

DataSculpt:通过多目标划分塑造长上下文 LLM 的数据景观

DataSculpt: Crafting Data Landscapes for Long-Context LLMs through Multi-Objective Partitioning

Peking University(北京大学);Baichuan Inc.(百川智能公司)

Keer Lu, Xiaonan Nie, Zheng Liang, Da Pan, Shusen Zhang, Keshi Zhao, Weipeng Chen, Zenan Zhou, Guosheng Dong, Bin Cui, Wentao Zhang

AI总结:

本文提出 DataSculpt 框架,将训练数据组织建模为多目标组合优化问题,通过语义聚类与簇内贪心拼接构建长上下文窗口,显著提升多项长上下文任务表现。

摘要:

近年来,大语言模型(LLM)在多种任务上展现出显著提升,其中之一便是长上下文能力。提升长上下文性能的关键在于有效的数据组织与管理策略,这些策略需整合来自多个领域的数据,并在训练期间优化上下文窗口。通过大量实验分析,我们发现设计有效数据管理策略时存在三大关键挑战,这些策略需使模型获得长上下文能力,同时不牺牲其他任务的性能:(1)多个领域的长文档不足;(2)上下文窗口的有效构建;(3)大规模数据集的高效组织。为应对这些挑战,我们提出 DataSculpt,这是一种专为长上下文训练设计的新型数据管理框架。我们首先将训练数据的组织形式化为一个多目标组合优化问题,重点关注相关性、同质性、完整性和效率等属性。具体而言,我们的方法采用由粗到细的流程,以高效且有效地优化训练数据组织。我们首先基于语义相似性对数据进行聚类(粗粒度),随后在每个聚类内执行多目标贪心搜索,对文档进行评分并拼接成各种上下文窗口(细粒度)。综合评估表明,DataSculpt 显著增强了长上下文训练性能,使检索增强提升 18.09%、摘要提升 21.23%、阅读理解提升 21.27%、代码补全提升 3.81%,同时保持模型的整体能力,并带来 4.88% 的提升。

英文摘要:

In recent years, Large Language Models (LLMs) have demonstrated significant improvements across a variety of tasks, one of which is the long-context capability. The key to improving long-context performance lies in effective data organization and management strategies that integrate data from multiple domains and optimize the context window during training. Through extensive experimental analysis, we identified three key challenges in designing effective data management strategies that enable the model to achieve long-context capability without sacrificing performance in other tasks: (1) a shortage of long documents across multiple domains, (2) effective construction of context windows, and (3) efficient organization of large-scale datasets. To address these challenges, we introduce DataSculpt, a novel data management framework designed for long-context training. We first formulate the organization of training data as a multi-objective combinatorial optimization problem, focusing on attributes including relevance, homogeneity, integrity, and efficiency. Specifically, our approach utilizes a coarse-to-fine methodology to optimize training data organization both efficiently and effectively. We begin by clustering the data based on semantic similarity (coarse), followed by a multi-objective greedy search within each cluster to score and concatenate documents into various context windows (fine). Our comprehensive evaluations demonstrate that DataSculpt significantly enhances long-context training performance, resulting in improvements of 18.09% in retrieval augmentation, 21.23% in summarization, 21.27% in reading comprehension, and a 3.81% increase in code completion, while also maintaining overall model proficiency with a 4.88% improvement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑