arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.16802cs.CLcs.LG
收藏

AutoPSV:自动化过程监督验证器

AutoPSV: Automated Process-Supervised Verifier

The University of Hong Kong(香港大学);The Chinese University of Hong Kong(香港中文大学);University of Cambridge(剑桥大学);University of Edinburgh(爱丁堡大学);City University of Hong Kong(香港城市大学)

Jianqiao Lu, Zhiyang Dou, Hongru Wang, Zeyu Cao, Jianbo Dai, Yingjia Wan, Zhijiang Guo

AI总结:

本文提出 AutoPSV,利用仅依据最终答案正确性训练的验证模型产生步骤级置信度变化并自动标注推理错误,显著提升了多候选答案选择性能。

摘要:

在本研究中,我们提出了一种名为 Automated Process-Supervised Verifier(自动化过程监督验证器,AutoPSV)的新方法,通过自动标注推理步骤来增强大语言模型(LLM)的推理能力。AutoPSV 首先围绕最终答案的正确性训练一个验证模型,使其能够生成自动化过程标注。该验证模型会为每个推理步骤分配一个置信度分数,表示从该步骤开始继续推理直至得到正确最终答案的概率。我们检测各推理步骤之间验证置信度分数的相对变化,以自动标注推理过程,从而即使在没有标准答案的场景下也能实现错误检测。这缓解了对大量人工标注的需求,也降低了模型诱导标注方法所伴随的高昂计算成本。我们通过实验验证,基于最终答案正确性训练得到的验证模型所学习到的步骤级置信度变化,能够有效识别推理步骤中的错误。我们证明,当验证模型使用 AutoPSV 生成的过程标注进行训练后,其在从多个 LLM 生成输出中选择正确答案方面表现出更好的性能。值得注意的是,我们在数学推理和常识推理的五个数据集上均取得了显著提升。AutoPSV 的源代码可在 https://github.com/rookie-joe/AutoPSV 获取。

英文摘要:

In this work, we propose a novel method named \textbf{Auto}mated \textbf{P}rocess-\textbf{S}upervised \textbf{V}erifier (\textbf{\textsc{AutoPSV}}) to enhance the reasoning capabilities of large language models (LLMs) by automatically annotating the reasoning steps. \textsc{AutoPSV} begins by training a verification model on the correctness of final answers, enabling it to generate automatic process annotations. This verification model assigns a confidence score to each reasoning step, indicating the probability of arriving at the correct final answer from that point onward. We detect relative changes in the verification's confidence scores across reasoning steps to automatically annotate the reasoning process, enabling error detection even in scenarios where ground truth answers are unavailable. This alleviates the need for numerous manual annotations or the high computational costs associated with model-induced annotation approaches. We experimentally validate that the step-level confidence changes learned by the verification model trained on the final answer correctness can effectively identify errors in the reasoning steps. We demonstrate that the verification model, when trained on process annotations generated by \textsc{AutoPSV}, exhibits improved performance in selecting correct answers from multiple LLM-generated outputs. Notably, we achieve substantial improvements across five datasets in mathematics and commonsense reasoning. The source code of \textsc{AutoPSV} is available at \url{https://github.com/rookie-joe/AutoPSV}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑