arXiv 2507.127592025-07-17cs.CLcs.AI

对数几率算术无需训练即可激发长推理能力

Logit Arithmetic Elicits Long Reasoning Capabilities Without Training

Computer Science and Engineering University of Michigan（计算机科学与工程系密歇根大学）

Yunxiang Zhang, Muhammad Khalifa, Lechen Zhang, Xin Liu, Ayoung Lee, Xinliang Frederick Zhang, Farima Fatahi Bayat, Lu Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出解码时方法ThinkLogit及其DPO变体，利用小模型引导大模型进行长推理，无需或仅需极少额外训练即可显著提升数学推理的pass@1性能。

摘要：

大型推理模型（LRMs）能够通过包含回溯和自我纠正等认知策略的长链思维（CoT）进行复杂推理。近期研究表明，一些模型本身具备这些长推理能力，但可能需要通过额外训练来解锁。我们的工作首先探究是否能在不进行任何训练的情况下激发这种行为。为此，我们提出了一种解码时方法 ThinkLogit，该方法利用对数几率算术（Liu et al., 2024），以一个规模显著更小的模型作为引导器（guider），来调节目标大型语言模型以进行长推理。随后我们表明，可以通过对从目标模型和引导器模型中采样的正确/错误推理对进行偏好优化来训练引导器模型，从而进一步提升性能——我们将这一设置称为 ThinkLogit-DPO。我们的实验表明，在四个数学数据集上，当使用比目标模型小21倍的 R1-Distill-Qwen-1.5B 作为引导器来引导 Qwen2.5-32B 时，ThinkLogit 和 ThinkLogit-DPO 分别实现了 pass@1 相对提升26%和29%。最后，我们证明 ThinkLogit 能够迁移通过强化学习获得的长推理技能，与 Qwen2.5-32B 基础模型相比，pass@1 相对提升13%。我们的工作提出了一种计算高效的方法，以极少或无需额外训练的方式激发大型模型的长推理能力。

英文摘要：

Large reasoning models (LRMs) can do complex reasoning via long chain-of-thought (CoT) involving cognitive strategies such as backtracking and self-correction. Recent studies suggest that some models inherently possess these long reasoning abilities, which may be unlocked via extra training. Our work first investigates whether we can elicit such behavior without any training. To this end, we propose a decoding-time approach, ThinkLogit, which utilizes logits arithmetic (Liu et al., 2024) to tune a target large LM for long reasoning using a substantially smaller model as guider. We then show that we can further boost performance by training the guider model with preference optimization over correct/incorrect reasoning pairs sampled from both the target and guider model -- a setup we refer to as ThinkLogit-DPO. Our experiments demonstrate that ThinkLogit and ThinkLogit-DPO achieve a relative improvement in pass@1 by 26% and 29%, respectively, over four mathematical datasets using the Qwen2.5-32B when guided by R1-Distill-Qwen-1.5B -- a model 21x smaller. Lastly, we show that ThinkLogit can transfer long reasoning skills acquired through reinforcement learning, improving pass@1 by 13% relative compared to the Qwen2.5-32B base model. Our work presents a computationally-efficient method to elicit long reasoning in large models with minimal or no additional training.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。