arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.00544cs.LGcs.CL
收藏

逃离模式抽彩:多响应训练提升语言模型泛化能力

Escaping the Mode Lottery: Multi-Response Training Improves Language Model Generalization

Department of Computer Science, Purdue University(计算机科学系,普渡大学)

Hasan Amin, Kian Ahrabian, Ming Yin, Rajiv Khanna

AI总结:

本文提出多响应训练(MRT)方法,通过保留每个提示的多个有效响应来缓解传统单响应微调导致的“模式抽彩”问题,并从统计角度揭示了其提升分布泛化的原理和适用条件。

摘要:

现代语言模型微调通常为每个提示配对单个响应,尽管许多提示允许多个有效补全。这实际上将多模态条件分布简化为单样本视图,我们称之为“模式抽彩”现象,其中训练强调一部分合理模式而忽略其他模式。我们研究了多响应训练(MRT),该方法保留每个提示的多个响应,并建立了关于何时以及为何有帮助的原则性解释。我们的关键见解是,提示和响应是不同的统计资源:额外的提示减少输入分布的不确定性,而额外的响应减少条件输出分布的不确定性。这产生了方差-预算权衡,预测了何时保留多个响应是有价值的,显示了随着提示级不确定性占主导地位而收益递减,并解释了为什么大型冗余语料库可以表现出隐式的多响应效应。我们进一步分析了响应选择,并表明Random-K-of-N是分布微调的无偏默认选择,基于奖励的选择可能导致模式坍缩,而子模质量-多样性目标提供了一种具有理论保证的高效替代方案。受控模拟验证了预测的方差和选择效应,包括一个惊人的失败模式,其中仅奖励选择产生的梯度与真实目标不一致。在结构化和真实世界数据集上,包括一个新的多提示、多响应基准,MRT一致地改善了分布泛化,在响应多样性高、提示冗余性低的场景中收益最大。MRT将响应多重性重新定义为数据分配问题,并提供了明确的指导:当响应廉价且多样时,保留多个响应不是启发式方法,而是基于统计的选择。

英文摘要:

Modern language-model fine-tuning typically pairs each prompt with a single response, even though many prompts admit multiple valid completions. This effectively reduces a multi-modal conditional distribution to a one-sample view, a phenomenon we call the "mode lottery," where training emphasizes a subset of plausible modes while leaving others underrepresented. We study multi-response training (MRT), which retains multiple responses per prompt, and develop a principled account of when and why it helps. Our key insight is that prompts and responses are distinct statistical resources: additional prompts reduce uncertainty about the input distribution, while additional responses reduce uncertainty about the conditional output distribution. This yields a variance-budget tradeoff that predicts when retaining multiple responses is worthwhile, shows diminishing returns as prompt-level uncertainty dominates, and explains why large redundant corpora can exhibit an implicit multi-response effect. We further analyze response selection, and show that Random-K-of-N is the unbiased default for distributional fine-tuning, reward-based selection can induce mode collapse, and a submodular quality-diversity objective provides an efficient alternative with theoretical guarantees. Controlled simulations validate the predicted variance and selection effects, including a striking failure mode where reward-only selection produces gradients misaligned with the true objective. Across structured and real-world datasets, including a new multi-prompt, multi-response benchmark, MRT consistently improves distributional generalization, with the largest gains in high response-diversity, low prompt-redundancy regimes. MRT reframes response multiplicity as a data-allocation problem with clear guidance: when responses are cheap and diverse, keeping more than one is not a heuristic, but a statistically grounded choice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑