arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.00761cs.CL
收藏

FIRE:数据质量评级灵活集成以实现高效预训练

FIRE: Flexible Integration of Data Quality Ratings for Effective Pre-Training

Peking University(北京大学);Beihang University(北航);Tsinghua University(清华大学);Meituan(美团)

Liangyu Xu, Xuemiao Zhang, Feiyu Duan, Sirui Wang, Rongxiang Weng, Jingang Wang, Xunliang Cai

AI总结:

提出FIRE框架,集成多个数据质量评估器并采用渐进式选择方案,以少于随机基线37.5%的数据显著提升LLM预训练性能。

摘要:

选择高质量数据可以提高大型语言模型(LLM)的预训练效率。现有方法通常依赖启发式技术或单一质量信号,限制了其全面评估数据质量的能力。在这项工作中,我们提出了FIRE,一个灵活且可扩展的框架,用于集成多个数据质量评估器,从而能够跨多个维度对数据质量进行全面评估。FIRE将多个质量信号对齐到统一空间中,并集成多样化的数据质量评估器,为每个数据点提供全面的质量信号。此外,我们引入了一种基于FIRE的渐进式数据选择方案,该方案迭代地优化高质量数据点的选择。大量实验表明,FIRE优于其他数据选择方法,并在广泛的下游任务中显著提升预训练模型性能,同时达到目标性能所需的训练数据量不到随机基线所需数据的37.5%。

英文摘要:

Selecting high-quality data can improve the pretraining efficiency of large language models (LLMs). Existing methods generally rely on heuristic techniques or single quality signals, limiting their ability to evaluate data quality comprehensively. In this work, we propose FIRE, a flexible and scalable framework for integrating multiple data quality raters, which allows for a comprehensive assessment of data quality across various dimensions. FIRE aligns multiple quality signals into a unified space, and integrates diverse data quality raters to provide a comprehensive quality signal for each data point. Further, we introduce a progressive data selection scheme based on FIRE that iteratively refines the selection of high-quality data points. Extensive experiments show that FIRE outperforms other data selection methods and significantly boosts pretrained model performance across a wide range of downstream tasks, while requiring less than 37.5\% of the training data needed by the Random baseline to reach the target performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑