用于对齐语言模型推理能力的自精炼指令微调
Self-Refine Instruction-Tuning for Aligning Reasoning in Language Models
AI总结：
提出自精炼指令微调方法，通过两阶段过程（LLM演示指令微调与基于DPO的自精炼）对齐小模型与大模型的推理能力，在常识和数学推理任务上显著超越传统指令微调。
摘要：
较小与较大语言模型之间推理能力的对齐，主要通过使用强大大型语言模型（LLMs）生成的演示进行监督微调（SFT）来实现。尽管这些方法提供了性能更强的模型，但由于训练仅依赖于所提供的演示，它们未能展现出足够强的泛化能力。在本文中，我们提出了自精炼指令微调方法，促使较小语言模型自精炼其能力。我们的方法基于两阶段过程：首先，利用LLMs提供的演示进行指令微调，在LLMs和小语言模型（SLMs）之间转移推理能力；然后，受指导的模型通过偏好优化策略自精炼其能力。特别是，第二阶段基于直接偏好优化算法运行精炼启发式机制，通过对生成的回复进行自动采样并使用来自LLMs的真实值提供奖励，引导SLMs输出一系列推理路径。在常识和数学推理任务上获得的结果表明，该方法在领域内和领域外场景中均显著优于指令微调，对齐了较小和较大语言模型的推理能力。
英文摘要：
The alignments of reasoning abilities between smaller and larger Language Models are largely conducted via Supervised Fine-Tuning (SFT) using demonstrations generated from robust Large Language Models (LLMs). Although these approaches deliver more performant models, they do not show sufficiently strong generalization ability as the training only relies on the provided demonstrations. In this paper, we propose the Self-refine Instruction-tuning method that elicits Smaller Language Models to self-refine their abilities. Our approach is based on a two-stage process, where reasoning abilities are first transferred between LLMs and Small Language Models (SLMs) via Instruction-tuning on demonstrations provided by LLMs, and then the instructed models Self-refine their abilities through preference optimization strategies. In particular, the second phase operates refinement heuristics based on the Direct Preference Optimization algorithm, where the SLMs are elicited to deliver a series of reasoning paths by automatically sampling the generated responses and providing rewards using ground truths from the LLMs. Results obtained on commonsense and math reasoning tasks show that this approach significantly outperforms Instruction-tuning in both in-domain and out-domain scenarios, aligning the reasoning abilities of Smaller and Larger Language Models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
