arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.14526cs.CLcs.AI
收藏

基于聚类的语言模型训练均衡数据采样

Balanced Data Sampling for Language Model Training with Clustering

School of Computer Science, Fudan University(复旦大学计算机科学学院);Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学)

Yunfan Shao, Linyang Li, Zhaoye Fei, Hang Yan, Dahua Lin, Xipeng Qiu

AI总结:

针对大型语言模型训练中随机采样忽略数据分布不均衡的问题,提出ClusterClip Sampling方法,通过聚类平衡常见与稀有样本并引入重复截断缓解过拟合,实验证明其优于随机采样等基线方法。

摘要:

数据在大型语言模型(LLMs)的训练中起着基础性作用。尽管人们已经关注数据集的收集与组合,但确定训练中的数据采样策略仍是一个悬而未决的问题。大多数LLMs采用简单的随机采样策略进行训练。然而,这种采样策略忽略了训练数据分布不均衡的特性,可能是次优的。在本文中,我们提出ClusterClip Sampling,以平衡训练数据的文本分布,从而实现更好的模型训练。具体而言,ClusterClip Sampling利用数据聚类来反映训练集的数据分布,并基于聚类结果在训练期间平衡常见样本和稀有样本。引入了重复截断操作,以缓解由某些簇的样本导致的过拟合问题。大量实验验证了ClusterClip Sampling的有效性,在各种训练数据集和大型语言模型下,其性能优于随机采样和其他基于聚类的采样变体。

英文摘要:

Data plays a fundamental role in the training of Large Language Models (LLMs). While attention has been paid to the collection and composition of datasets, determining the data sampling strategy in training remains an open question. Most LLMs are trained with a simple strategy, random sampling. However, this sampling strategy ignores the unbalanced nature of training data distribution, which can be sub-optimal. In this paper, we propose ClusterClip Sampling to balance the text distribution of training data for better model training. Specifically, ClusterClip Sampling utilizes data clustering to reflect the data distribution of the training set and balances the common samples and rare samples during training based on the cluster results. A repetition clip operation is introduced to mitigate the overfitting issue led by samples from certain clusters. Extensive experiments validate the effectiveness of ClusterClip Sampling, which outperforms random sampling and other cluster-based sampling variants under various training datasets and large language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑