寻找RELIFF:通过信念工程塑造推理行为而无需推理监督
Finding RELIEF: Shaping Reasoning Behavior without Reasoning Supervision via Belief Engineering
AI总结:
本文提出RELIFF,通过信念工程无需推理监督塑造大型推理模型的行为,实验表明其在效率和忠实度任务上表现优异。
摘要:
大型推理模型(LRMs)在复杂问题解决中取得了显著成功,但它们往往面临计算冗余或推理不忠实的问题。当前塑造LRM行为的方法通常依赖于强化学习或使用黄金标准推理轨迹的微调,这种范式不仅计算成本高,而且难以扩展。在本文中,我们揭示LRMs具有潜在的推理信念,这些信念内部跟踪其自身的推理特征,可以通过简单的logit探测来捕捉。基于这一见解,我们提出了推理信念工程(RELIFF),一个简单而有效的框架,通过将模型的自我概念与目标信念蓝图对齐来塑造LRM行为。关键的是,RELIFF完全 bypass 了需要推理轨迹监督的需求。它通过在合成的、自我反思的问题回答对上进行微调来内化期望的特征。在效率和忠实度任务上的广泛实验表明,RELIFF在要求较低的训练成本的同时,能够匹配或超越行为监督和偏好基线。进一步分析验证了改变模型的推理信念能有效塑造其实际行为。
英文摘要:
Large reasoning models (LRMs) have achieved remarkable success in complex problem-solving, yet they often suffer from computational redundancy or reasoning unfaithfulness. Current methods for shaping LRM behavior typically rely on reinforcement learning or fine-tuning with gold-standard reasoning traces, a paradigm that is both computationally expensive and difficult to scale. In this paper, we reveal that LRMs possess latent \textit{reasoning beliefs} that internally track their own reasoning traits, which can be captured through simple logit probing. Building upon this insight, we propose Reasoning Belief Engineering (RELIEF), a simple yet effective framework that shapes LRM behavior by aligning the model's self-concept with a target belief blueprint. Crucially, RELIEF completely bypasses the need for reasoning-trace supervision. It internalizes desired traits by fine-tuning on synthesized, self-reflective question-answering pairs that affirm the target belief. Extensive experiments on efficiency and faithfulness tasks demonstrate that RELIEF matches or outperforms behavior-supervised and preference-based baselines while requiring lower training costs. Further analysis validates that shifting a model's reasoning belief effectively shapes its actual behavior.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。