Training large language models (LLMs) to spend more time thinking and reflection before responding is crucial for effectively solving complex reasoning tasks in fields such as science, coding, and mathematics. However, the effectiveness of mechanisms like self-reflection and self-correction depends on the model's capacity to accurately assess its own performance, which can be limited by factors such as initial accuracy, question difficulty, and the lack of external feedback. In this paper, we delve into a two-player paradigm that separates the roles of reasoning and critique models, where the critique model provides step-level feedback to supervise the reasoning (actor) model during both test-time and train-time. We first propose AutoMathCritique, an automated and scalable framework for collecting critique data, resulting in a dataset of $76,321$ responses paired with step-level feedback. Fine-tuning language models with this dataset enables them to generate natural language feedback for mathematical reasoning. We demonstrate that the critique models consistently improve the actor's performance on difficult queries at test-time, especially when scaling up inference-time computation. Motivated by these findings, we introduce the critique-based supervision to the actor's self-training process, and propose a critique-in-the-loop self-improvement method. Experiments show that the method improves the actor's exploration efficiency and solution diversity, especially on challenging queries, leading to a stronger reasoning model. Lastly, we take the preliminary step to explore training self-talk reasoning models via critique supervision and showcase its potential. Our code and datasets are at \href{https://mathcritique.github.io/}{https://mathcritique.github.io/}.
Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization
Zhejiang University（浙江大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；South China University of Technology（华南理工大学）；University of British Columbia（不列颠哥伦比亚大学）；Fudan University（复旦大学）；University of Hong Kong（香港大学）
DeepCritic: Deliberate Critique with Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学首都人工智能学院）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）
Self-Evolving Critique Abilities in Large Language Models
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Shenzhen International Center for Industrial and Applied Mathematics（深圳国际工业与应用数学中心）；Qwen Team, Alibaba Inc.（阿里云通义团队）
We fine-tune large language models to write natural language critiques (natural language critical comments) using behavioral cloning. On a topic-based summarization task, critiques written by our models help humans find flaws in summaries that they would have otherwise missed. Our models help find naturally occurring flaws in both model and human written summaries, and intentional flaws in summaries written by humans to be deliberately misleading. We study scaling properties of critiquing with both topic-based summarization and synthetic tasks. Larger models write more helpful critiques, and o…
A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Tasks
DJCAD, University of Dundee, United Kingdom（邓迪大学DJCAD部门）；ARG-tech, SSEN, University of Dundee, United Kingdom（邓迪大学）；School of Computer Science, University of Birmingham, United Kingdom（伯明翰大学计算机科学学院）
Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning
The Hong Kong University of Science and Technology（香港科学与技术大学）；University of Electronic Science and Technology of China（电子科技大学）；Dalian University of Technology（大连理工大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Zhejiang University（浙江大学）；University of Oxford（牛津大学）；NVIDIA（NVIDIA公司）；University of Surrey（萨里大学）
LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；Pattern Recongnition Center, WechatAI, Tencent Inc（图案识别中心，微信AI，腾讯公司）；Dept. of Comp. Sci. & Tech., Tsinghua University（计算机科学与技术系，清华大学）
Training Language Model to Critique for Better Refinement
China Telecom Research Institute（中国电信研究院）；The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University（清华大学对话人工智能（CoAI）小组）；University of Electronic Science and Technology of China（电子科技大学）；The Knowledge Engineering Group (KEG), Tsinghua University（清华大学知识工程小组）；Zhipu AI（智谱AI）
Scalable Oversight for Superhuman AI via Recursive Self-Critiquing
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）；University of Chinese Academy of Sciences（中国科学院大学）
RefCritic: Training Long Chain-of-Thought Critic Models with Refinement Feedback
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）；Alibaba Group（阿里巴巴集团）；University of Chinese Academy of Sciences（中国科学院大学）
In-context Demonstration Matters: On Prompt Optimization for Pseudo-Supervision Refinement
Center for Advanced Intelligence Project, RIKEN（RIKEN先进智能项目中心）；Northeastern University（东北大学）；UNC-Chapel Hill（北卡罗来纳大学教堂山分校）；Center for Advanced Intelligence Project, RIKEN Graduate School of Frontier Sciences, The University of Tokyo（RIKEN前沿科学研究生院、东京大学先进智能项目中心）
Enhancing Agentic Textual Graph Retrieval with Synthetic Stepwise Supervision
Institute for AI Industry Research (AIR), Tsinghua University（人工智能产业研究院（AIR），清华大学）；Peking University（北京大学）；School of Information, Renmin University of China（中国人民大学信息学院）；Harbin Institute of Technology（哈尔滨工业大学）；North China Electric Power University（华北电力大学）；GDS Holdings Limited（GDS控股有限公司）
KG-TRACES: Enhancing Large Language Models with Knowledge Graph-constrained Trajectory Reasoning and Attribution Supervision
Zhejiang University（浙江大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Fudan University（复旦大学）
Enhancing Trustworthy Clinical Diagnosis Decision-Making in Large Language Models via Etiology-Aware Attention Supervision
Zhejiang University（浙江大学）；University of Chinese Academy of Sciences（中国科学院大学）；First Affiliated Hospital, College of Medicine, Zhejiang University（浙江大学医学院附属第一医院）
Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision
School of Computer Science, University of Sheffield（谢菲尔德大学计算机科学学院）；School of Electronic Engineering and Computer Science, Queen Mary University of London（伦敦大学玛丽女王学院电子工程与计算机科学学院）；The Alan Turing Institute（艾伦·图灵研究所）
