通过细粒度数据采样的目标感知语言建模
Target-Aware Language Modeling via Granular Data Sampling
AI总结:
本研究提出基于多粒度n-gram特征的重要性采样方法,通过低维特征选择大规模预训练数据,使语言模型在特定领域表现优异且不损害其他任务性能,仅需约1%数据即可达到完整数据集效果。
摘要:
语言模型预训练通常面向广泛的应用场景,并融合来自不同来源的数据。然而,在某些情况下,我们期望模型在特定领域表现出色,同时不显著影响其他领域的性能。一种低成本且直接的方法是利用低维数据特征进行采样,从而为特定领域的应用场景选择大规模预训练数据。在本研究中,我们重新审视了基于包含多粒度token的n-gram特征的重要性采样,该方法在句子压缩与表示能力之间取得了良好的平衡。我们观察到,采样数据与目标下游任务性能高度相关,同时保持了在其他任务上的有效性。这促成了一种新的数据采样范式,使语言模型能够在选定的文档上更高效地进行预训练。在八个基准测试中,我们证明仅需约1%的数据,预训练模型即可达到完整RefinedWeb数据的性能,并在125M至1.5B的模型规模下优于随机选取的样本。
英文摘要:
Language model pretraining generally targets a broad range of use cases and incorporates data from diverse sources. However, there are instances where we desire a model that excels in specific areas without markedly compromising performance in other areas. A cost-effective and straightforward approach is sampling with low-dimensional data features, which allows to select large-scale pretraining data for domain-specific use cases. In this work, we revisit importance sampling with n-gram features consisting of multi-granular tokens, which strikes a good balance between sentence compression and representation capabilities. We observed the sampled data to have a high correlation with the target downstream task performance while preserving its effectiveness on other tasks. This leads to the proposed data sampling paradigm where language models can be pretrained more efficiently on selected documents. On eight benchmarks we demonstrate with $\sim$1% of the data, pretrained models perform on par with the full RefinedWeb data and outperform randomly selected samples for model sizes ranging from 125M to 1.5B.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。