推理还是捏造：通过提示锚定的成对聚合实现无捷径推理
To Reason or to Fabricate: Reasoning Without Shortcuts via Hint-Anchored Pairwise Aggregation
AI总结：
针对强化学习前数据重叠导致模型利用捷径记忆答案的问题，提出HIPPO框架，通过提示注入触发重叠行为并利用成对奖励模型区分真实推理与捷径合理化，显著提升推理真实性及泛化能力。
摘要：
虽然强化学习显著增强了LLM的推理能力，但其有效性受到预训练前数据重叠的严重削弱，即RL数据集与预训练或SFT语料库重叠，导致模型通过记忆正确答案和编造事后推理来利用捷径。为了解决这个问题，我们引入了HIPPO，一种新颖的RL框架，它集成了提示注入聚合和定制的成对奖励模型。通过利用提示注入故意触发重叠诱导的行为，生成的轨迹自然地作为成对比较的显式锚点。这提供了高度可区分的偏好信号，使得轻量级评判模型能够可靠地区分真正的推理演绎与捷径驱动的合理化，而成对公式确保了与标准PRM相比稳定且鲁棒的优化。大量实验表明，HIPPO在标准基线上产生了显著改进，并有效泛化到分布外的一般任务，表明它提取了真实的、可迁移的推理技能，而不是表面的捷径模式。
英文摘要：
While reinforcement learning (RL) significantly enhances LLM reasoning, its efficacy is severely undermined by Pre-RL data overlap, where RL datasets overlap with pretraining or SFT corpora, causing models to exploit shortcuts by memorizing correct answers and fabricating post-hoc reasoning. To address this, we introduce HIPPO, a novel RL framework that integrates hint-injected aggregation with a tailored pairwise reward model. By utilizing hint injection to deliberately trigger overlap-induced behaviors, the resulting traces naturally serve as explicit anchors for pairwise comparison. This provides highly discriminable preference signals, enabling a lightweight judge model to reliably distinguish genuine reasoning deduction from shortcut-driven rationalization, while the pairwise formulation ensures stable and robust optimization compared to standard PRMs. Extensive experiments demonstrate that HIPPO yields substantial improvements over standard baselines and generalizes effectively to out-of-distribution general tasks, showing it extracts authentic, transferable reasoning skills rather than superficial shortcut patterns.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
