一种更健康的模型：法国ModernBERT的案例研究
A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT
AI总结：
本文通过比较多样性驱动和随机采样的预训练方法，发现多样性驱动方法在某些任务上能提升性能，同时减少预训练数据规模。
摘要：
多样性近年来在自然语言处理社区中引起了越来越多的关注。同时，最先进的transformer模型如ModernBERT使用非常大的预训练数据集，这些数据集主要是由规模而非多样性驱动。这促使我们研究多样性对ModernBERT预训练的影响。我们在此研究中进行此类调查，旨在减少预训练数据集的规模，同时至少保持可比的性能。我们比较了多样性驱动的采样算法，以选择最佳的一种。我们发现，在某些任务中，多样性驱动的采样相对于随机采样的预训练数据在相同规模下可以提高10分。我们还发现，一个在1.5亿个token的多样性驱动数据集上预训练483小时的模型，可以产生与在24亿个token的随机驱动数据集上预训练1775小时的模型相媲美的性能。
英文摘要：
Diversity has been gaining interest in the NLP community in recent years. At the same time, state-of-the-art transformer models such as ModernBERT use very large pre-training datasets, which are driven by size rather than by diversity. This summons to investigate the impact of diversity on pre-training. We do so in this study, with the express intent of reducing pre-training dataset size, while retaining at least comparable performance. We compare diversity-driven sampling algorithms, and we use the best one to pre-train several ModernBERT models on French with a fixed compute budget. We fine-tune and evaluate them on a variety of French benchmarks. We compare them with models pre-trained on randomly sampled data of commensurate size, with the same compute budget. We find that both random and diversity-driven sampling may reduce the pre-training dataset by up to 94% and the pre-training time by up to 73% while maintaining performance. Moreover, in some tasks, the inherent quality of models, estimated via head-only fine-tuning, is up to 10 points higher with diversity sampling than with random sampling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
