arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.07154cs.LGcs.AI
收藏

重新思考扩展测试时间计算时的微调:限制置信度可提升数学推理

Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning

Stanford University(斯坦福大学);University of Michigan(密歇根大学)

Feng Chen, Allan Raventos, Nan Cheng, Surya Ganguli, Shaul Druckmann

AI总结:

通过限制模型置信度改进数学推理,研究发现传统交叉熵损失与测试时间策略pass@N存在不一致,提出改进的训练损失以提升性能。

摘要:

最近在大语言模型(LLM)中的进展突显了通过扩展测试时间计算来在复杂任务上取得强大性能的威力,例如数学推理和代码生成。这提出了一个关键问题:如何修改模型训练以在随后的测试时间计算策略和预算下优化性能?为了探讨这一点,我们聚焦于pass@N,一种简单的测试时间策略,它在N个独立样本中搜索正确答案。我们发现,令人惊讶的是,使用交叉熵(CE)损失进行训练可能会与pass@N不一致,因为pass@N的准确性随着训练时间的增加而下降。我们解释了这种不一致的根源在于CE诱导的模型过度自信,并通过实验验证了过度自信作为通过pass@N扩展测试时间计算的阻碍因素的预测。此外,我们建议了一种改进的训练损失,它通过限制模型置信度来更好地与pass@N对齐,从而恢复pass@N的测试性能。我们的算法在多个场景下展示了在MATH和MiniF2F基准上的改进数学推理能力:(1)回答数学问题;(2)通过搜索不同形状的证明树来证明定理。总体而言,我们的工作强调了共同设计LLM开发的两个传统独立阶段的重要性:训练时间协议和测试时间搜索和推理策略。

英文摘要:

Recent progress in large language models (LLMs) highlights the power of scaling test-time compute to achieve strong performance on complex tasks, such as mathematical reasoning and code generation. This raises a critical question: how should model training be modified to optimize performance under a subsequent test-time compute strategy and budget? To explore this, we focus on pass@N, a simple test-time strategy that searches for a correct answer in $N$ independent samples. We show, surprisingly, that training with cross-entropy (CE) loss can be ${\it misaligned}$ with pass@N in that pass@N accuracy ${\it decreases}$ with longer training. We explain the origins of this misalignment in terms of model overconfidence induced by CE, and experimentally verify our prediction of overconfidence as an impediment to scaling test-time compute via pass@N. Furthermore we suggest a principled, modified training loss that is better aligned to pass@N by limiting model confidence and rescuing pass@N test performance. Our algorithm demonstrates improved mathematical reasoning on MATH and MiniF2F benchmarks under several scenarios: (1) providing answers to math questions; and (2) proving theorems by searching over proof trees of varying shapes. Overall our work underscores the importance of co-designing two traditionally separate phases of LLM development: training-time protocols and test-time search and reasoning strategies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑