arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.14565cs.LGcs.CL
收藏

ReVISE:通过内在自验证在测试时学习精炼

ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification

Yonsei University(延世大学)

Hyunseok Lee, Seunghyuk Oh, Jaehyung Kim, Jinwoo Shin, Jihoon Tack

AI总结:

提出ReVISE框架,通过在线偏好学习和课程学习使大模型在测试时进行内在自验证与推理纠正,实现高效自纠错并显著提升推理性能。

摘要:

自我意识,即评估和纠正自身生成的能力,是人类智能的基本方面,这使得其在大型语言模型(LLMs)中的复制成为一项重要且具有挑战性的任务。以往的工作通过采用大量的强化学习或依赖大型外部验证器来解决这一问题。在这项工作中,我们提出了通过内在自验证进行精炼(Refine via Intrinsic Self-Verification,ReVISE),这是一个高效且有效的框架,使LLMs能够通过自验证来纠正其输出。ReVISE的核心思想是使LLMs能够验证其推理过程,并基于其验证不断重新思考推理轨迹。我们引入了基于在线偏好学习的结构化课程来高效地实现这一点。具体来说,由于ReVISE涉及两个具有挑战性的任务(即自验证和推理纠正),我们使用课程学习依次处理每个任务,收集失败和成功的推理路径以构建用于高效训练的偏好对。在推理过程中,我们的方法通过整合自验证和纠正能力,享受自然的测试时扩展,并通过我们提出的置信度感知解码机制进一步增强。我们在各种推理任务上的实验表明,ReVISE实现了高效的自纠正,并显著提高了推理性能。

英文摘要:

Self-awareness, i.e., the ability to assess and correct one's own generation, is a fundamental aspect of human intelligence, making its replication in large language models (LLMs) an important yet challenging task. Previous works tackle this by employing extensive reinforcement learning or rather relying on large external verifiers. In this work, we propose Refine via Intrinsic Self-Verification (ReVISE), an efficient and effective framework that enables LLMs to self-correct their outputs through self-verification. The core idea of ReVISE is to enable LLMs to verify their reasoning processes and continually rethink reasoning trajectories based on its verification. We introduce a structured curriculum based upon online preference learning to implement this efficiently. Specifically, as ReVISE involves two challenging tasks (i.e., self-verification and reasoning correction), we tackle each task sequentially using curriculum learning, collecting both failed and successful reasoning paths to construct preference pairs for efficient training. During inference, our approach enjoys natural test-time scaling by integrating self-verification and correction capabilities, further enhanced by our proposed confidence-aware decoding mechanism. Our experiments on various reasoning tasks demonstrate that ReVISE achieves efficient self-correction and significantly improves reasoning performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑