arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.01170cs.LGcs.AIcs.CLstat.APstat.ML
收藏

在线推理校准:测试时训练使可泛化的符合性大语言模型推理

Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning

Department of Electrical Engineering and Computer Science (MIT EECS)(麻省理工学院电气工程与计算机科学系);Computer Science and Artificial Intelligence Laboratory (MIT CSAIL)(麻省理工学院计算机科学与人工智能实验室);Laboratory for Information and Decision Systems (MIT LIDS)(麻省理工学院信息与决策系统实验室);Computational Science and Engineering (MIT CSE)(麻省理工学院计算科学与工程);Massachusetts Institute of Technology(麻省理工学院)

Cai Zhou, Zekai Wang, Menghua Wu, Qianyu Julie Zhu, Flora C. Shi, Chenyu Wang, Ashia Wilson, Tommi Jaakkola, Stephen Bates

AI总结:

本文提出ORCA框架,通过测试时训练和符合性预测校准推理过程,提升大语言模型在分布变化下的效率和泛化能力,实验证明其在不同任务中具有更高的性能。

摘要:

尽管测试时扩展使大型语言模型能够解决高度困难的任务,但最先进的结果以极高的计算成本为代价。这些低效可以归因于后训练语言模型的校准错误以及流行采样技术中校准的缺乏。本文提出了在线推理校准(ORCA),一种校准采样过程的框架,借鉴了符合性预测和测试时训练。具体来说,我们引入了一种元学习过程,为每个输入更新校准模块。这使我们能够在分布偏移下提供有效的置信度估计,例如在不同推理阶段出现的思维模式或模型开发与部署之间的提示分布中。ORCA不仅在符合性风险上提供了理论保证,而且在不同推理任务中表现出更高的效率和泛化能力。在风险水平δ=0.1时,ORCA在分布内任务中使用监督标签时提高了Qwen2.5-32B的效率,节省高达47.5%,使用自一致性标签时节省40.7%。在零样本跨域设置下,它将MATH-500的节省从静态校准基线的24.8%提高到67.0%,同时保持低的实证错误率,且在模型家族和下游基准测试中呈现相同趋势。我们的代码在https://github.com/wzekai99/ORCA上公开可用。

英文摘要:

While test-time scaling has enabled large language models to solve highly difficult tasks, state-of-the-art results come at exorbitant compute costs. These inefficiencies can be attributed to the miscalibration of post-trained language models, and the lack of calibration in popular sampling techniques. Here, we present Online Reasoning Calibration (ORCA), a framework for calibrating the sampling process that draws upon conformal prediction and test-time training. Specifically, we introduce a meta-learning procedure that updates the calibration module for each input. This allows us to provide valid confidence estimates under distributional shift, e.g. in thought patterns that occur across different stages of reasoning, or in prompt distributions between model development and deployment. ORCA not only provides theoretical guarantees on conformal risks, but also empirically shows higher efficiency and generalization across different reasoning tasks. At risk level $δ=0.1$, ORCA improves Qwen2.5-32B efficiency on in-distribution tasks with savings up to 47.5% with supervised labels and 40.7% with self-consistency labels. Under zero-shot out-of-domain settings, it improves MATH-500 savings from 24.8% of the static calibration baseline to 67.0% while maintaining a low empirical error rate, and the same trend holds across model families and downstream benchmarks. Our code is publicly available at https://github.com/wzekai99/ORCA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑