arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.08310cs.CL
收藏

ScalingFilter:通过反向利用缩放定律评估数据质量

ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws

University of Science and Technology of China(中国科学技术大学);Microsoft Research Asia(微软亚洲研究院);Tsinghua University(清华大学);Southeast University(东南大学)

Ruihang Li, Yixuan Wei, Miaosen Zhang, Nenghai Yu, Han Hu, Houwen Peng

AI总结:

针对现有数据质量过滤依赖参考数据集易引入偏差、损害多样性的问题,提出基于双模型困惑度差异的ScalingFilter方法,反向利用缩放定律评估数据质量,在下游任务性能与语义多样性间实现最优平衡。

摘要:

高质量数据对大语言模型的预训练性能至关重要。遗憾的是,现有质量过滤方法依赖已知的高质量数据集作为参考,这可能引入潜在偏差并损害多样性。本文提出ScalingFilter,这是一种基于在相同数据上训练的两个语言模型之间的困惑度差异来评估文本质量的新方法,从而消除了过滤过程中参考数据集的影响。理论分析表明,ScalingFilter等价于对缩放定律的反向利用。通过在经多种质量过滤器处理的相同数据源上训练1.3B参数的模型,我们发现ScalingFilter能够提升预训练模型在下游任务中的零样本性能。为评估质量过滤引入的偏差,我们提出了语义多样性指标,该指标利用文本嵌入模型进行语义表示。大量实验表明,语义多样性是数据集多样性的可靠衡量指标,且ScalingFilter在下游性能与语义多样性之间实现了最优平衡。

英文摘要:

High-quality data is crucial for the pre-training performance of large language models. Unfortunately, existing quality filtering methods rely on a known high-quality dataset as reference, which can introduce potential bias and compromise diversity. In this paper, we propose ScalingFilter, a novel approach that evaluates text quality based on the perplexity difference between two language models trained on the same data, thereby eliminating the influence of the reference dataset in the filtering process. An theoretical analysis shows that ScalingFilter is equivalent to an inverse utilization of scaling laws. Through training models with 1.3B parameters on the same data source processed by various quality filters, we find ScalingFilter can improve zero-shot performance of pre-trained models in downstream tasks. To assess the bias introduced by quality filtering, we introduce semantic diversity, a metric of utilizing text embedding models for semantic representations. Extensive experiments reveal that semantic diversity is a reliable indicator of dataset diversity, and ScalingFilter achieves an optimal balance between downstream performance and semantic diversity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑