BOLT：无需蒸馏即可在语言模型中引导长链思维
BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillation
AI总结：
本文提出 BOLT 方法，通过上下文学习引导、监督微调和在线训练三阶段，从标准指令模型出发，无需蒸馏 o1 类模型或人工标注即可赋予 LLM 长链思维推理能力，并在多个基准上验证了其有效性。
摘要：
大型语言模型（LLM），如 OpenAI 的 o1，已展现出卓越的推理能力。o1 在回答问题之前会生成一条长链思维（LongCoT）。LongCoT 使 LLM 能够有效分析问题、制定计划、反思和回溯。这些行为使 LLM 能够解决复杂问题。在 o1 发布后，许多团队尝试复现其 LongCoT 和推理能力。在方法上，他们主要依赖知识蒸馏，使用来自具备 LongCoT 能力的现有模型（例如 OpenAI-o1、Qwen-QwQ、DeepSeek-R1-Preview）的数据，这给系统性地开发此类推理能力留下了重大不确定性。在数据领域方面，这些工作主要集中于数学，少数包含编码，限制了其泛化能力。本文提出了一种新方法，无需从 o1 类模型蒸馏或昂贵的人工标注即可赋予 LLM 的 LongCoT 能力，其中我们从一个标准指令模型引导 LongCoT（BOLT）。BOLT 包含三个阶段：1）在标准指令模型上通过上下文学习进行 LongCoT 数据引导；2）LongCoT 监督微调；3）在线训练以进一步优化 LongCoT 能力。在 BOLT 中，引导阶段仅需构建少量上下文示例；在我们的实验中，我们创建了 10 个示例，证明了该方法的可行性。我们使用 Llama-3.1-70B-Instruct 来引导 LongCoT，并将我们的方法应用于不同模型规模（7B、8B、70B）。我们在多个基准测试上取得了令人印象深刻的表现，包括 Arena-Hard、MT-Bench、WildBench、ZebraLogic、MATH500，这些基准评估了多样化的任务解决和推理能力。
英文摘要：
Large language models (LLMs), such as o1 from OpenAI, have demonstrated remarkable reasoning capabilities. o1 generates a long chain-of-thought (LongCoT) before answering a question. LongCoT allows LLMs to analyze problems, devise plans, reflect, and backtrack effectively. These actions empower LLM to solve complex problems. After the release of o1, many teams have attempted to replicate its LongCoT and reasoning capabilities. In terms of methods, they primarily rely on knowledge distillation with data from existing models with LongCoT capacities (e.g., OpenAI-o1, Qwen-QwQ, DeepSeek-R1-Preview), leaving significant uncertainties on systematically developing such reasoning abilities. In terms of data domains, these works focus narrowly on math while a few others include coding, limiting their generalizability. This paper introduces a novel approach to enable LLM's LongCoT capacity without distillation from o1-like models or expensive human annotations, where we bootstrap LongCoT (BOLT) from a standard instruct model. BOLT involves three stages: 1) LongCoT data bootstrapping with in-context learning on a standard instruct model; 2) LongCoT supervised finetuning; 3) online training to further refine LongCoT capacities. In BOLT, only a few in-context examples need to be constructed during the bootstrapping stage; in our experiments, we created 10 examples, demonstrating the feasibility of this approach. We use Llama-3.1-70B-Instruct to bootstrap LongCoT and apply our method to various model scales (7B, 8B, 70B). We achieve impressive performance on a variety of benchmarks, Arena-Hard, MT-Bench, WildBench, ZebraLogic, MATH500, which evaluate diverse task-solving and reasoning capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
