arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.06457cs.LGcs.AIcs.CL
收藏

V-STaR: 为自学推理器训练验证器

V-STaR: Training Verifiers for Self-Taught Reasoners

Mila, Université de Montréal（米拉-蒙特利尔大学）；Microsoft Research（微软研究院）；University of Edinburgh（爱丁堡大学）；Google Deepmind（谷歌DeepMind）

Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, Rishabh Agarwal

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有大语言模型自我改进方法丢弃错误解、浪费信息的问题，提出V-STaR方法，用自改进中的正误解通过DPO训练验证器以在推理时选优解，多轮迭代后在相关基准上准确率提升4%至17%。

摘要：

大语言模型（LLM）常见的自我改进方法，例如STaR（自学推理器），会在自生成的解决方案上迭代微调LLM以提升其问题求解能力。然而，这类方法会丢弃此过程中生成的大量错误解决方案，可能忽略了这些解决方案中的宝贵信息。为弥补这一缺陷，我们提出V-STaR方法，该方法利用自我改进过程中生成的正确与错误解决方案，通过DPO（直接偏好优化）训练一个验证器，用于判断模型生成解决方案的正确性。该验证器在推理阶段用于从多个候选解决方案中选出一个。运行多轮V-STaR可逐步得到更优的推理器和验证器，在使用LLaMA2模型的常见代码生成与数学推理基准测试中，其测试准确率相较现有自我改进与验证方法提升了4%至17%。

英文摘要：

Common self-improvement approaches for large language models (LLMs), such as STaR, iteratively fine-tune LLMs on self-generated solutions to improve their problem-solving ability. However, these approaches discard the large amounts of incorrect solutions generated during this process, potentially neglecting valuable information in such solutions. To address this shortcoming, we propose V-STaR that utilizes both the correct and incorrect solutions generated during the self-improvement process to train a verifier using DPO that judges correctness of model-generated solutions. This verifier is used at inference time to select one solution among many candidate solutions. Running V-STaR for multiple iterations results in progressively better reasoners and verifiers, delivering a 4% to 17% test accuracy improvement over existing self-improvement and verification approaches on common code generation and math reasoning benchmarks with LLaMA2 models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑