arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.09151cs.CL
收藏

UniMax:面向大规模多语言预训练的更公平且更有效的语言采样方法

UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining

Google Research(谷歌研究院)

Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, Orhan Firat

AI总结:

本文提出UniMax采样方法,通过限制低资源语言重复次数并均匀覆盖高资源语言,在多语言预训练中优于传统温度采样,并发布29万亿字符的mC4语料库及umT5模型检查点。

摘要:

预训练的多语言大语言模型通常使用基于启发式温度的采样方法来平衡不同语言之间的数据分布。然而,以往的工作尚未在不同模型规模下系统评估不同预训练语言分布的有效性。本文提出了一种新的采样方法 UniMax,该方法在更均匀地覆盖高资源语言的同时,通过显式限制每种语言语料库的重复次数来缓解对低资源语言的过拟合。我们在多个多语言基准测试上进行了大量消融实验,测试了一系列采样策略,并同时改变模型规模。我们发现 UniMax 优于标准的基于温度的采样方法,并且随着模型规模的增大,其优势持续保持。作为贡献的一部分,我们发布了:(i)一个经过改进和更新的 mC4 多语言语料库,包含 107 种语言、总计 29 万亿字符;(ii)一套使用 UniMax 采样训练的 umT5 预训练模型检查点。

英文摘要:

Pretrained multilingual large language models have typically used heuristic temperature-based sampling to balance between different languages. However previous work has not systematically evaluated the efficacy of different pretraining language distributions across model scales. In this paper, we propose a new sampling method, UniMax, that delivers more uniform coverage of head languages while mitigating overfitting on tail languages by explicitly capping the number of repeats over each language's corpus. We perform an extensive series of ablations testing a range of sampling strategies on a suite of multilingual benchmarks, while varying model scale. We find that UniMax outperforms standard temperature-based sampling, and the benefits persist as scale increases. As part of our contribution, we release: (i) an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters across 107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained with UniMax sampling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑