arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.03181cs.CL
收藏

微调于多样化推理链驱动大语言模型推理内思维链精炼

Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs

Ubiquitous Knowledge Processing Lab (UKP Lab), TU Darmstadt and ATHENE National Research Center for Applied Cybersecurity, Germany(图宾根大学知识处理实验室(UKP Lab)及德国应用网络安全国家研究中心ATHENE);Dept. of ECE & Ingenuity Labs Research Institute, Queen’s University, Canada(皇后大学电子工程系及创新实验室研究机构);University of Bath, UK(巴斯大学)

Haritz Puerto, Tilek Chubakov, Xiaodan Zhu, Harish Tayyar Madabushi, Iryna Gurevych

AI总结:

本文提出微调LLM在单次推理内生成多样化思维链(DCoT),实现推理链的自我精炼,提升多任务性能,尤其对数值答案任务效果显著。

摘要:

要求大型语言模型(LLM)生成中间推理步骤,即思维链(CoT),已被证明是提升性能的有效方法。以往的方法侧重于生成多个独立的思维链,通过集成或其他事后策略将其组合以增强推理能力。在本工作中,我们提出了一种新颖的方法,即对LLM进行微调,使其在单次推理步骤内生成一系列多样化思维链(DCoT),这与先前主要基于并行思维链生成的工作有根本不同。DCoT使LLM能够在无需外部反馈的情况下,获得在推理过程中对推理链进行精炼的能力。通过一系列涵盖多种推理类型任务的严格实验,我们表明,在DCoT上微调相较于CoT基线,在不同模型家族和规模(1.3B至70B)上均提升了性能。这些改进对于具有较大结果状态空间的任务(如涉及数值答案的任务)尤为显著。我们的工作意义重大,因为定量分析和人工评估均显示,观察到的增益源于模型在同一推理步骤内生成第二条改进链来精炼初始推理链的能力,展示了此前难以捉摸的自我改进能力。我们的代码和数据公开于https://github.com/UKPLab/acl2025-diverse-cot。

英文摘要:

Requiring a large language model (LLM) to generate intermediary reasoning steps, known as Chain of Thought (CoT), has been shown to be an effective way of boosting performance. Previous approaches have focused on generating multiple independent CoTs, combining them through ensembling or other post-hoc strategies to enhance reasoning. In this work, we introduce a novel approach where LLMs are fine-tuned to generate a sequence of Diverse Chains of Thought (DCoT) within a single inference step, which is fundamentally different from prior work that primarily operate on parallel CoT generations. DCoT allows LLMs to gain the ability to perform within-inference refinement of reasoning chains without requiring external feedback. Through a rigorous set of experiments spanning a wide range of tasks that require various reasoning types, we show that fine-tuning on DCoT improves performance over the CoT baseline across model families and scales (1.3B to 70B). These improvements are particularly impactful for tasks with a large result state space, such as those involving numeric answers. Our work is also significant because both quantitative analyses and manual evaluations reveal the observed gains stem from the models' ability to refine an initial reasoning chain by generating a second, improved chain within the same inference step, demonstrating previously elusive self-improvement. Our code and data are publicly available at https://github.com/UKPLab/acl2025-diverse-cot.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑