Recent advancements in test time compute, particularly through the use of verifier models, have significantly enhanced the reasoning capabilities of Large Language Models (LLMs). This generator-verifier approach closely resembles the actor-critic framework in reinforcement learning (RL). However, current verifier models in LLMs often rely on supervised fine-tuning without temporal difference learning such as Q-learning. This paper introduces VerifierQ, a novel approach that integrates Offline Q-learning into LLM verifier models. We address three key challenges in applying Q-learning to LLMs: (1) handling utterance-level Markov Decision Processes (MDPs), (2) managing large action spaces, and (3) mitigating overestimation bias. VerifierQ introduces a modified Bellman update for bounded Q-values, incorporates Implicit Q-learning (IQL) for efficient action space management, and integrates a novel Conservative Q-learning (CQL) formulation for balanced Q-value estimation. Our method enables parallel Q-value computation and improving training efficiency. While recent work has explored RL techniques like MCTS for generators, VerifierQ is among the first to investigate the verifier (critic) aspect in LLMs through Q-learning. This integration of RL principles into verifier models complements existing advancements in generator techniques, potentially enabling more robust and adaptive reasoning in LLMs. Experimental results on mathematical reasoning tasks demonstrate VerifierQ's superior performance compared to traditional supervised fine-tuning approaches, with improvements in efficiency, accuracy and robustness. By enhancing the synergy between generation and evaluation capabilities, VerifierQ contributes to the ongoing evolution of AI systems in addressing complex cognitive tasks across various domains.
Logic-Regularized Verifier Elicits Reasoning from LLMs
Department of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术系）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））
SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning
The State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）；School of Computer Science, Beijing University of Posts and Telecommunications（北京邮电大学计算机学院）
DARL: Encouraging Diverse Answers for General Reasoning without Verifiers
School of Informatics, Xiamen University（厦门大学信息学院）；Kuaishou Technology（快手科技）；Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism（福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室（厦门大学），文化和旅游部）
STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning
Generative AI Lab, College of Computing and Data Science, Nanyang Technological University, Singapore（生成式人工智能实验室，计算与数据科学学院，南洋理工大学，新加坡）；Tongyi Lab, Alibaba Group（通义实验室，阿里巴巴集团）
Coupled Variational Reinforcement Learning for Language Model General Reasoning
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Train Long, Think Short: Curriculum Learning for Efficient Reasoning
King Abdullah University of Science and Technology (KAUST), Saudi Arabia（卡布尔大学科学与技术大学）；Massachusetts Institute of Technology (MIT), Cambridge, MA, USA（麻省理工学院）；Princeton University, Princeton, NJ, USA（普林斯顿大学）
VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models
Zhejiang University（浙江大学）；Meituan Group（美团集团）；Peking University（北京大学）；University of Electronic Science and Technology of China（电子科技大学）；The Hong Kong University of Science and Technology（香港科技大学）
