通过认知对齐增强小型LLM的推理能力
Enhancing Reasoning Abilities of Small LLMs with Cognitive Alignment
AI总结：
本文提出CRV系统和CogPO算法，通过多智能体批判、重思与验证思维链，并将推理过程与小型模型认知能力对齐，有效提升了小型大型推理模型的推理能力。
摘要：
大型推理模型（LRMs），如OpenAI的o1和DeepSeek-R1，通过深度思考取得了显著的推理能力提升。然而，这些提升伴随着巨大的资源需求，凸显了训练有效的小型推理模型的必要性。一个关键挑战是，与大型模型相比，小型模型具有不同的推理能力和认知轨迹。因此，直接从大型LRMs向小型模型蒸馏思维链（CoT）解释有时会无效，且通常需要大量标注数据。在本文中，我们首先引入了一个新颖的Critique-Rethink-Verify（CRV）系统，旨在训练更小但强大的LRMs。我们的CRV系统由多个LLM智能体组成，每个智能体专注于独特任务：(i) 根据小型模型的认知能力批判CoT解释，(ii) 基于批判重新思考并细化这些CoT，以及(iii) 验证细化结果的正确性。在CRV系统的基础上，我们进一步提出了认知偏好优化（CogPO）算法，通过将推理过程与认知能力对齐来持续增强小型模型的推理能力。在具有挑战性的推理基准上的全面评估证明了我们的CRV+CogPO框架的有效性，其性能大幅超越其他方法。
英文摘要：
The reasoning capabilities of large reasoning models (LRMs), such as OpenAI's o1 and DeepSeek-R1, have seen substantial advancements through deep thinking. However, these enhancements come with significant resource demands, underscoring the need for training effective small reasoning models. A critical challenge is that small models possess different reasoning capacities and cognitive trajectories compared with their larger counterparts. Hence, directly distilling chain-of-thought (CoT) rationales from large LRMs to smaller ones can sometimes be ineffective and often requires a substantial amount of annotated data. In this paper, we first introduce a novel Critique-Rethink-Verify (CRV) system, designed for training smaller yet powerful LRMs. Our CRV system consists of multiple LLM agents, each specializing in unique tasks: (i) critiquing the CoT rationales according to the cognitive capabilities of smaller models, (ii) rethinking and refining these CoTs based on the critiques, and (iii) verifying the correctness of the refined results. Building on the CRV system, we further propose the Cognitive Preference Optimization (CogPO) algorithm to continuously enhance the reasoning abilities of smaller models by aligning their reasoning processes with their cognitive capacities. Comprehensive evaluations on challenging reasoning benchmarks demonstrate the efficacy of our CRV+CogPO framework, which outperforms other methods by a large margin.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
