arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.21189cs.LGcs.AI
收藏

为何Pass@k优化会损害Pass@1:LLM微调中的提示干扰

Prompt Interference in LLM Post-Training: Why Pass@k Optimization Can Degrade Pass@1

Singapore University of Technology and Design(新加坡科技设计大学);University of Maryland, College Park(马里兰大学学院公园分校);University of Central Florida(佛罗里达中央大学)

Anas Barakat, Souradip Chakraborty, Khushbu Pahwa, Amrit Singh Bedi

AI总结:

该研究揭示了在LLM微调中,pass@k优化可能损害pass@1的原因,通过梯度冲突和提示干扰机制,提出了理论解释并验证了实验结果。

摘要:

Pass@k是用于可验证的大语言模型任务(包括数学推理、代码生成和简答推理)中广泛使用的性能度量标准,它定义为当任意k个独立采样的解决方案通过验证器时视为成功。这种多样本推理度量标准激励了推理感知的微调方法,这些方法直接优化pass@$k$。然而,先前的工作报告了一个反复出现的权衡:在这样的方法下,pass@k会提高而pass@1会下降。这种权衡在实践中很重要,因为pass@1往往仍然是一个困难的操作约束,由于延迟和成本预算、不完美的验证器覆盖以及需要可靠的单次 fallback 的需求。我们研究了这种权衡的起源,并提供了理论上的表征,说明在梯度冲突由提示干扰引起的情况下,pass@k策略优化如何通过重新加权提示向低成功率提示倾斜来减少pass@1。我们展示了pass@$k$策略梯度可以与pass@1梯度冲突,因为pass@$k$优化隐式地将提示重新加权向低成功率提示;当这些提示被称为负干扰时,它们的加权会将pass@k更新方向旋转远离pass@1方向。我们通过在可验证的数学推理任务上进行大型语言模型实验来说明我们的理论发现。

英文摘要:

Pass@$k$ is a widely used performance metric for verifiable large language model tasks, including mathematical reasoning, code generation, and short-answer reasoning. It defines success if any of $k$ independently sampled solutions passes a verifier. This multi-sample inference metric has motivated inference-aware fine-tuning methods that directly optimize pass@$k$. However, prior work reports a trade-off: pass@$k$ improves while pass@1 degrades under such methods. This trade-off is practically important because pass@1 often remains a hard operational constraint due to latency and cost budgets, imperfect verifier coverage, and the need for a reliable single-shot fallback. We study the origin of this trade-off and provide a theoretical characterization of when pass@$k$ policy optimization can reduce pass@1 through gradient conflict induced by prompt interference. This interference arises because shared model parameters couple prompts, so improving performance on some prompts can adversely affect others. We show that pass@$k$ policy gradients can conflict with pass@1 gradients because pass@$k$ optimization implicitly reweights prompts toward low-success prompts; when these prompts are negatively interfering, their upweighting can rotate the pass@$k$ update direction away from the pass@1 direction. We illustrate our findings with large language model experiments on verifiable mathematical reasoning and coding tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑