arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.09170cs.CL
收藏

通过自适应思考从大型语言模型中蒸馏推理能力

Distilling Reasoning Ability from Large Language Models with Adaptive Thinking

College of Computer Science and Technology, National University of Defense Technology(计算机科学与技术学院,国防科技大学);College of Intelligence Science and Technology, National University of Defense Technology(智能科学与技术学院,国防科技大学)

Xiaoshu Chen, Sihang Zhou, Ke Liang, Xinwang Liu

AI总结:

针对思维链微调中预思考机制对推理错误敏感的问题,提出后思考机制及自适应思考机制,整合两者优点,在12个推理任务上验证了有效性。

摘要:

思维链微调(cot-finetuning)旨在赋予小型语言模型(SLM)推理能力,通过允许其模仿大型语言模型(LLM)的推理过程而不仅仅是预测答案,从而提升其在特定任务上的表现。大多数现有的思维链微调方法采用预思考机制,允许SLM在提供答案之前生成推理过程。该机制使SLM能够分析和思考复杂问题,但也使答案的正确性对推理过程中的微小错误高度敏感。因此,我们提出了一种稳健的后思考机制,即在推理过程之前生成答案。得益于这种答案优先的设置:1)答案可以免受推理过程中微小错误的不利影响;2)推理过程作为答案的错误放大器,使SLM专注于学习困难样本;3)推理效率也能从该设置中受益,因为在推理时,用户可以在答案输出后立即停止生成。然而,尽管后思考机制带来了诸多优势并提升了SLM在特定任务上的整体性能,但与预思考机制相比,它可能丧失思考问题并将复杂问题分解为简单子问题的能力。因此,借助软提示调优,提出了一种即插即用的自适应思考机制,以整合预思考机制和后思考机制的优点,其中引入了一个感知模块,通过感知问题的复杂性来自适应地提示SLM先回答或先思考。在12个推理任务和2个代表性语言模型上进行了大量实验,以证明所提出机制的有效性。

英文摘要:

Chain of thought finetuning (cot-finetuning) aims to endow small language models (SLM) with reasoning ability to improve their performance towards specific tasks by allowing them to imitate the reasoning procedure of large language models (LLM) beyond simply predicting the answers. Most existing cot-finetuning methods adopt a pre-thinking mechanism, allowing the SLM to generate a rationale before providing an answer. This mechanism enables SLM to analyze and think about complex questions, but it also makes answer correctness highly sensitive to minor errors in rationale. Therefore, we propose a robust post-thinking mechanism to generate answers before rationale. Thanks to this answer-first setting, 1) the answer can escape from the adverse effects caused by minor errors in the rationale; 2) the rationale serves as an error amplifier to the answer, which makes the SLM focus on learning hard samples; 3) the inferring efficiency can also benefit from the setting since users can stop the generation right after answers are outputted when inference is conducted. However, although the post-thinking mechanism brings many advantages and improves the overall performance of SLM on specific tasks, it may lose the ability to think about the questions and decompose complex questions into simple sub-questions compared to pre-thinking mechanism. Therefore, a plug-and-play adaptive-thinking mechanism is proposed with the aid of the soft prompt tuning to integrate the merits of the pre-thinking mechanism and post-thinking mechanism, in which a perception module is introduced to adaptively prompt SLM answer or think first based on perceiving the complexity of the questions. Extensive experiments are conducted across 12 reasoning tasks and 2 representative language models to demonstrate the effectiveness of the proposed mechanism.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑