arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.13752cs.AIcs.CL
收藏

寻找RELIFF:通过信念工程塑造推理行为而无需推理监督

Finding RELIEF: Shaping Reasoning Behavior without Reasoning Supervision via Belief Engineering

Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系);Sun Yat-sen University(中山大学);Zhejiang University(浙江大学);WUST(华中科技大学)

Chak Tou Leong, Dingwei Chen, Heming Xia, Qingyu Yin, Sunbowen Lee, Jian Wang, Wenjie Li

AI总结:

本文提出RELIFF,通过信念工程无需推理监督塑造大型推理模型的行为,实验表明其在效率和忠实度任务上表现优异。

摘要:

大型推理模型(LRMs)在复杂问题解决中取得了显著成功,但它们往往面临计算冗余或推理不忠实的问题。当前塑造LRM行为的方法通常依赖于强化学习或使用黄金标准推理轨迹的微调,这种范式不仅计算成本高,而且难以扩展。在本文中,我们揭示LRMs具有潜在的推理信念,这些信念内部跟踪其自身的推理特征,可以通过简单的logit探测来捕捉。基于这一见解,我们提出了推理信念工程(RELIFF),一个简单而有效的框架,通过将模型的自我概念与目标信念蓝图对齐来塑造LRM行为。关键的是,RELIFF完全 bypass 了需要推理轨迹监督的需求。它通过在合成的、自我反思的问题回答对上进行微调来内化期望的特征。在效率和忠实度任务上的广泛实验表明,RELIFF在要求较低的训练成本的同时,能够匹配或超越行为监督和偏好基线。进一步分析验证了改变模型的推理信念能有效塑造其实际行为。

英文摘要:

Large reasoning models (LRMs) have achieved remarkable success in complex problem-solving, yet they often suffer from computational redundancy or reasoning unfaithfulness. Current methods for shaping LRM behavior typically rely on reinforcement learning or fine-tuning with gold-standard reasoning traces, a paradigm that is both computationally expensive and difficult to scale. In this paper, we reveal that LRMs possess latent \textit{reasoning beliefs} that internally track their own reasoning traits, which can be captured through simple logit probing. Building upon this insight, we propose Reasoning Belief Engineering (RELIEF), a simple yet effective framework that shapes LRM behavior by aligning the model's self-concept with a target belief blueprint. Crucially, RELIEF completely bypasses the need for reasoning-trace supervision. It internalizes desired traits by fine-tuning on synthesized, self-reflective question-answering pairs that affirm the target belief. Extensive experiments on efficiency and faithfulness tasks demonstrate that RELIEF matches or outperforms behavior-supervised and preference-based baselines while requiring lower training costs. Further analysis validates that shifting a model's reasoning belief effectively shapes its actual behavior.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑