超越固定长度:Bucket预训练即所需
Beyond Fixed Length: Bucket Pre-training is All You Need
AI总结:
针对固定长度预训练数据策略的缺陷,提出基于填充率、截断率和拼接率的多桶数据组合方法,自适应组织数据以提升LLM预训练效率与效果。
摘要:
大型语言模型(LLMs)在各类任务中表现优异,预训练阶段是其能力的基石。然而,传统固定长度的预训练数据组合策略存在实际挑战。使用较短序列时,文档常被截断,可能导致信息丢失并影响模型捕捉长距离依赖的能力。相反,较长序列需拼接多个文档,这会引入噪声,影响文档自然边界与语义连贯性,且需大量计算开销。为应对这些挑战,我们首先建立三个评估数据组合质量的定量指标:填充率、截断率和拼接率。基于这些指标,我们提出一种新颖的多桶数据组合方法,超越固定长度范式。该方法自适应组织训练数据,以所提指标衡量实现最优组合质量,为预训练提供更灵活高效的方式。我们进行了广泛实验,结果表明所提方法显著提升了LLM预训练的效率与有效性。
英文摘要:
Large Language Models (LLMs) have demonstrated exceptional performance across various tasks, with pre-training stage serving as the cornerstone of their capabilities. However, the conventional fixed-length data composition strategy for pre-training presents several practical challenges. When using shorter sequences, documents are often truncated, potentially leading to information loss and affecting the model's ability to capture long-range dependencies. Conversely, longer sequences require concatenation of multiple documents, which can introduce noise and affect the natural document boundaries and semantic coherence as well as require substantial computational overhead. To address these challenges, we first establish three quantitative metrics for evaluating data composition quality: padding ratio, truncation ratio, and concatenation ratio. Building upon these metrics, we propose a novel multi-bucket data composition method that transcends the fixed-length paradigm. Our approach adaptively organizes training data to achieve optimal composition quality as measured by the proposed metrics, offering a more flexible and efficient approach for pre-training. We conduct extensive experiments and the results demonstrate that our proposed method significantly enhances both the efficiency and effectiveness of LLM pre-training.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。