棱柱形合成:基于梯度的数据多样化提升语言模型推理中的泛化能力
Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning
AI总结:
研究语言模型训练数据多样性对泛化的作用,提出基于梯度熵的G - Vendi指标,进而构建棱柱形合成框架生成多样合成数据,有效提升模型性能,在多个基准测试中表现优于依赖更大数据生成器的模型。
摘要:
语言模型中的有效泛化关键取决于训练数据的多样性。现有多样性指标常依赖与模型行为脱节的表面启发式方法,难以达成目标。为此研究何种训练数据多样性驱动语言模型泛化及如何衡量与增强它。通过超300次训练运行的大规模实证分析表明,数据多样性可有力预测语言模型推理中的泛化。引入G - Vendi指标,基于模型诱导梯度的熵量化多样性,表现优于其他方法。在此基础上提出棱柱形合成框架,通过针对梯度空间中代表性不足区域生成多样合成数据。实验结果显示,随着合成数据规模增加,棱柱形合成持续提升模型性能,在多个基准测试中显著优于依赖更大数据生成器的现有模型。
英文摘要:
Effective generalization in language models depends critically on the diversity of their training data. Yet existing diversity metrics often fall short of this goal, relying on surface-level heuristics that are decoupled from model behavior. This motivates us to ask: What kind of diversity in training data actually drives generalization in language models -- and how can we measure and amplify it? Through large-scale empirical analyses spanning over 300 training runs, carefully controlled for data scale and quality, we show that data diversity can be a strong predictor of generalization in LLM reasoning -- as measured by average model performance on unseen out-of-distribution benchmarks. We introduce G-Vendi, a metric that quantifies diversity via the entropy of model-induced gradients. Despite using a small off-the-shelf proxy model for gradients, G-Vendi consistently outperforms alternative measures, achieving strong correlation (Spearman's $ρ\approx 0.9$) with out-of-distribution (OOD) performance on both natural language inference (NLI) and math reasoning tasks. Building on this insight, we present Prismatic Synthesis, a framework for generating diverse synthetic data by targeting underrepresented regions in gradient space. Experimental results show that Prismatic Synthesis consistently improves model performance as we scale synthetic data -- not just on in-distribution test but across unseen, out-of-distribution benchmarks -- significantly outperforming state-of-the-art models that rely on 20 times larger data generator than ours. For example, PrismMath-7B, our model distilled from a 32B LLM, outperforms R1-Distill-Qwen-7B -- the same base model trained on proprietary data generated by 671B R1 -- on 6 out of 7 challenging benchmarks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。