超越内省：通过外部主义行为反馈强化思维
Beyond Introspection: Reinforcing Thinking via Externalist Behavioral Feedback
AI总结：
本文提出DRR框架，通过外部主义行为反馈提升LLM推理质量，无需修改基础模型。
摘要：
尽管推理时的思考过程使大型语言模型（LLMs）能够解决复杂问题，但扩展的思考过程可能不可靠或不一致，因为模型的概率性质，尤其是在知识边界附近。现有方法试图通过让模型批评自己的推理来做出修正。然而，这种自我批评继承了原始输出的相同偏见，即内省幻觉。超越这种内省，并受动物学核心方法的启发，我们提出了一种外部主义三步框架Distillation-Reinforcement-Reasoning（DRR）。而不是依赖模型的内省，DRR评估其可观察行为以提供纠正反馈。DRR首先蒸馏推理者的可观察行为痕迹，然后训练一个轻量级的外部判别模型（DM）。在推理时，该DM充当批评者，识别并拒绝可疑的推理步骤。这种外部反馈迫使LLM丢弃有缺陷的路径并探索替代方案，从而在不改变基础模型的情况下提高推理质量。在多个推理基准测试中，我们的框架显著优于突出的自我批评方法。受益于轻量级且无标注设计，DRR为提高LLMs在广泛范围内的推理可靠性提供了可扩展且适应性强的解决方案。
英文摘要：
While inference-time thinking allows Large Language Models (LLMs) to address complex problems, the extended thinking process can be unreliable or inconsistent because of the model's probabilistic nature, especially near its knowledge boundaries. Existing approaches attempt to mitigate this by having the model critique its own reasoning to make corrections. However, such self-critique inherits the same biases of the original output, known as the introspection illusion. Moving beyond such introspection and inspired by core methodologies in ethology, we propose an externalist three-step framework Distillation-Reinforcement-Reasoning (DRR). Rather than relying on a model's introspection, DRR evaluates its observable behaviors to provide corrective feedback. DRR first distills the reasoner's behavioral traces, then trains a lightweight, external Discriminative Model (DM). At inference time, this DM acts as a critic, identifying and rejecting suspicious reasoning steps. This external feedback compels the LLM to discard flawed pathways and explore alternatives, thereby enhancing reasoning quality without altering the base model. Experiments on multiple reasoning benchmarks show that our framework significantly outperforms prominent self-critique methods. Benefiting from a lightweight and annotation-free design, DRR offers a scalable and adaptable solution for improving the reliability of reasoning in a wide range of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
