监督微调(SFT)通常用于训练语言模型以模仿针对给定指令的标注回复。在本文中,我们提出了Critique Fine-Tuning(CFT),一种在推理任务上比SFT更有效的方法。受人类学习过程强调批判性思维、深度分析和细致理解的启发,CFT不再简单模仿正确回复,而是训练模型对噪声回复进行批判,这些特质常被标准SFT忽视。为验证CFT的有效性,我们构建了多个批判数据集(如WebInstruct、MetaMath、NuminaMath),其中GPT-4o作为教师模型生成形式为([查询;噪声回复],批判)的批判内容。在这些数据集上的实验表明,CFT在六个数学推理基准上始终比SFT高出4-10%,并在包括Qwen2.5、Qwen2.5-Math和DeepSeek-Math在内的不同基础模型上均有效。值得注意的是,我们的模型Qwen2.5-Math-CFT仅需在8 x H100上对50K样本训练1小时,却在大多数基准上匹配或超越了使用超过2M样本的强大竞争对手Qwen2.5-Math-Instruct。此外,它还匹配了SimpleRL的性能,后者是使用140倍计算量训练的DeepSeek-r1复制品。在IF_Eval和MT-Bench上的实验进一步表明,CFT能显著增强模型的通用生成和指令遵循能力,大幅超越Qwen2.5-Math-Instruct。消融研究表明,CFT对噪声回复来源和教师批判模型具有鲁棒性。这些发现凸显了CFT为提升语言模型推理能力提供了一种更有效的替代方案。
英文摘要:
Supervised Fine-Tuning (SFT) is commonly used to train language models to imitate annotated responses for given instructions. In this paper, we propose Critique Fine-Tuning (CFT), a method more effective than SFT for reasoning tasks. Instead of simply imitating correct responses, CFT trains models to critique noisy responses, inspired by human learning processes that emphasize critical thinking, deeper analysis, and nuanced understanding - traits often overlooked by standard SFT. To validate the effectiveness of CFT, we construct multiple critique datasets (e.g., WebInstruct, MetaMath, NuminaMath), where GPT-4o serves as the teacher to generate critiques in the form of ([query; noisy response], critique). Experiments on these datasets demonstrate that CFT consistently outperforms SFT by 4-10% across six mathematical reasoning benchmarks, and is effective across different base models including Qwen2.5, Qwen2.5-Math, and DeepSeek-Math. Notably, our model Qwen2.5-Math-CFT only requires 1 hour of training on 8 x H100 over the 50K examples, yet matches or outperforms strong competitors like Qwen2.5-Math-Instruct on most benchmarks, which use over 2M samples. Moreover, it matches the performance of SimpleRL, which is a DeepSeek-r1 replication trained with 140 x more compute. Experiments on IF_Eval and MT-Bench further demonstrate that CFT can significantly enhance the model's general generation and instruction-following capabilities, outperforming the Qwen2.5-Math-Instruct by a large margin. Ablation studies show that CFT is robust to noisy response sources and teacher critique models. These findings highlight that CFT offers a more effective alternative to advance the reasoning of language models.
Training Language Models to Critique With Multi-agent Feedback
Shanghai AI Laboratory(上海人工智能实验室);School of Medical Technology, Beijing Institute of Technology(北京理工大学医学技术学院);School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院)
Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning
The Hong Kong University of Science and Technology(香港科学与技术大学);University of Electronic Science and Technology of China(电子科技大学);Dalian University of Technology(大连理工大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);Zhejiang University(浙江大学);University of Oxford(牛津大学);NVIDIA(NVIDIA公司);University of Surrey(萨里大学)
RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning
Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
HCCL, The Chinese University of Hong Kong, Hong Kong, China(香港中文大学人工智能研究中心,香港,中国);University of Cambridge, Cambridge, United Kingdom(剑桥大学,剑桥,英国);MMLab, The Chinese University of Hong Kong, Hong Kong, China(香港中文大学人工智能实验室,香港,中国);Shanghai Artificial Intelligence Laboratory, Shanghai, China(上海人工智能实验室,上海,中国)
DeepCritic: Deliberate Critique with Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学首都人工智能学院);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院)
AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));State University of New York at Buffalo(纽约州立大学布法罗分校);Huawei International, Singapore(新加坡华为国际)