低资源语言建模中用于合成数据生成的对比解码
Contrastive Decoding for Synthetic Data Generation in Low-Resource Language Modeling
AI总结：
本文提出利用对比解码生成合成语料，与原始数据混合训练，提升低资源语言建模及下游任务性能，尤其增强推理能力。
摘要：
大型语言模型（LLMs）在海量文本数据上进行训练，人们担心此类数据的极限可能很快就会被触及。一个潜在的解决方案是在从LLMs中采样的合成数据上进行训练。在这项工作中，我们基于这一想法，研究了对比解码在生成合成语料库中的益处。在一个受控环境中，我们使用在同一原始语料库（包含1亿个单词）上训练的好模型与差模型之间的相对差异来实验采样语料库。通过放大性能更优模型的信号，我们创建了一个合成语料库，并将其与原始训练数据混合。我们的研究结果表明，在合成数据与真实数据的混合数据上进行训练，能够提高语言建模目标以及一系列下游任务的性能。特别是，我们观察到，使用来自对比解码的合成数据进行混合训练，有益于需要更多推理能力的任务，而来自传统采样的合成数据则更有助于依赖表面层面语言能力的任务。
英文摘要：
Large language models (LLMs) are trained on huge amounts of textual data, and concerns have been raised that the limits of such data may soon be reached. A potential solution is to train on synthetic data sampled from LLMs. In this work, we build on this idea and investigate the benefits of contrastive decoding for generating synthetic corpora. In a controlled setting, we experiment with sampling corpora using the relative difference between a good and bad model trained on the same original corpus of 100 million words. By amplifying the signal from a model that has better performance, we create a synthetic corpus and mix it with the original training data. Our findings show that training on a mixture of synthesized and real data improves performance on the language modeling objective and a range of downstream tasks. In particular, we see that training with a mix of synthetic data from contrastive decoding benefits tasks that require more reasoning skills, while synthetic data from traditional sampling helps more on tasks dependent on surface level linguistic capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
