小型语言模型需要强验证器来实现推理自校正
Small Language Models Need Strong Verifiers to Self-Correct Reasoning
AI总结：
本文研究小型语言模型的自校正推理能力，提出利用正确解答引导错误批评并筛选后微调的新流程，在多个推理数据集上提升性能，并强调强验证器对自校正效果至关重要。
摘要：
自校正已成为提升大型语言模型（LLMs）推理性能的一种有前景的解决方案，其中LLMs利用自生成的、能指出错误的批评意见来改进其解答。本研究探讨了小型（参数规模不超过13B）语言模型（LMs）是否能够在来自更强LLMs的极少输入条件下，在推理任务中具备自校正能力。我们提出了一种新流程，引导较小规模的LMs收集自校正数据，以支持自精炼能力的训练。首先，我们利用正确解答来引导模型对其错误回答进行批评。其次，生成的批评意见经过筛选后，通过解答精炼用于对自校正推理器进行监督微调。我们的实验结果表明，在涵盖数学和常识推理的五个数据集上，两个模型的自校正能力均有所提升；当与基于GPT-4的强验证器搭配使用时，性能提升显著，但在使用弱自验证器来判断何时进行校正时，也发现了局限性。
英文摘要：
Self-correction has emerged as a promising solution to boost the reasoning performance of large language models (LLMs), where LLMs refine their solutions using self-generated critiques that pinpoint the errors. This work explores whether small (<= 13B) language models (LMs) have the ability of self-correction on reasoning tasks with minimal inputs from stronger LMs. We propose a novel pipeline that prompts smaller LMs to collect self-correction data that supports the training of self-refinement abilities. First, we leverage correct solutions to guide the model in critiquing their incorrect responses. Second, the generated critiques, after filtering, are used for supervised fine-tuning of the self-correcting reasoner through solution refinement. Our experimental results show improved self-correction abilities of two models on five datasets spanning math and commonsense reasoning, with notable performance gains when paired with a strong GPT-4-based verifier, though limitations are identified when using a weak self-verifier for determining when to correct.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
