自我意识,即评估和纠正自身生成的能力,是人类智能的基本方面,这使得其在大型语言模型(LLMs)中的复制成为一项重要且具有挑战性的任务。以往的工作通过采用大量的强化学习或依赖大型外部验证器来解决这一问题。在这项工作中,我们提出了通过内在自验证进行精炼(Refine via Intrinsic Self-Verification,ReVISE),这是一个高效且有效的框架,使LLMs能够通过自验证来纠正其输出。ReVISE的核心思想是使LLMs能够验证其推理过程,并基于其验证不断重新思考推理轨迹。我们引入了基于在线偏好学习的结构化课程来高效地实现这一点。具体来说,由于ReVISE涉及两个具有挑战性的任务(即自验证和推理纠正),我们使用课程学习依次处理每个任务,收集失败和成功的推理路径以构建用于高效训练的偏好对。在推理过程中,我们的方法通过整合自验证和纠正能力,享受自然的测试时扩展,并通过我们提出的置信度感知解码机制进一步增强。我们在各种推理任务上的实验表明,ReVISE实现了高效的自纠正,并显著提高了推理性能。
英文摘要:
Self-awareness, i.e., the ability to assess and correct one's own generation, is a fundamental aspect of human intelligence, making its replication in large language models (LLMs) an important yet challenging task. Previous works tackle this by employing extensive reinforcement learning or rather relying on large external verifiers. In this work, we propose Refine via Intrinsic Self-Verification (ReVISE), an efficient and effective framework that enables LLMs to self-correct their outputs through self-verification. The core idea of ReVISE is to enable LLMs to verify their reasoning processes and continually rethink reasoning trajectories based on its verification. We introduce a structured curriculum based upon online preference learning to implement this efficiently. Specifically, as ReVISE involves two challenging tasks (i.e., self-verification and reasoning correction), we tackle each task sequentially using curriculum learning, collecting both failed and successful reasoning paths to construct preference pairs for efficient training. During inference, our approach enjoys natural test-time scaling by integrating self-verification and correction capabilities, further enhanced by our proposed confidence-aware decoding mechanism. Our experiments on various reasoning tasks demonstrate that ReVISE achieves efficient self-correction and significantly improves reasoning performance.
Verification and Refinement of Natural Language Explanations through LLM-Symbolic Theorem Proving
University of Manchester(曼彻斯特大学);Idiap Research Institute(Idiap研究所);National Biomarker Centre, CRUK-MI, University of Manchester(英国曼彻斯特大学英国癌症研究中心曼彻斯特研究所国家生物标志物中心)
RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Wuhan University of Science and Technology(武汉科技大学);National University of Singapore(新加坡国立大学);Northeastern University at Qinhuangdao(东北大学秦皇岛分校);BOSS Zhipin(BOSS直聘)