arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.17703cs.CL
收藏

Critique Fine-Tuning:学习批判比学习模仿更有效

Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate

University of Waterloo(滑铁卢大学);Carnegie Mellon University(卡内基梅隆大学);Vector Institute(向量研究院)

Yubo Wang, Xiang Yue, Wenhu Chen

AI总结:

本文提出Critique Fine-Tuning(CFT),通过训练模型批判噪声回复而非模仿正确回复,在多个数学推理基准上以更少数据和计算量显著超越SFT,并增强模型通用生成与指令遵循能力。

摘要:

监督微调(SFT)通常用于训练语言模型以模仿针对给定指令的标注回复。在本文中,我们提出了Critique Fine-Tuning(CFT),一种在推理任务上比SFT更有效的方法。受人类学习过程强调批判性思维、深度分析和细致理解的启发,CFT不再简单模仿正确回复,而是训练模型对噪声回复进行批判,这些特质常被标准SFT忽视。为验证CFT的有效性,我们构建了多个批判数据集(如WebInstruct、MetaMath、NuminaMath),其中GPT-4o作为教师模型生成形式为([查询;噪声回复],批判)的批判内容。在这些数据集上的实验表明,CFT在六个数学推理基准上始终比SFT高出4-10%,并在包括Qwen2.5、Qwen2.5-Math和DeepSeek-Math在内的不同基础模型上均有效。值得注意的是,我们的模型Qwen2.5-Math-CFT仅需在8 x H100上对50K样本训练1小时,却在大多数基准上匹配或超越了使用超过2M样本的强大竞争对手Qwen2.5-Math-Instruct。此外,它还匹配了SimpleRL的性能,后者是使用140倍计算量训练的DeepSeek-r1复制品。在IF_Eval和MT-Bench上的实验进一步表明,CFT能显著增强模型的通用生成和指令遵循能力,大幅超越Qwen2.5-Math-Instruct。消融研究表明,CFT对噪声回复来源和教师批判模型具有鲁棒性。这些发现凸显了CFT为提升语言模型推理能力提供了一种更有效的替代方案。

英文摘要:

Supervised Fine-Tuning (SFT) is commonly used to train language models to imitate annotated responses for given instructions. In this paper, we propose Critique Fine-Tuning (CFT), a method more effective than SFT for reasoning tasks. Instead of simply imitating correct responses, CFT trains models to critique noisy responses, inspired by human learning processes that emphasize critical thinking, deeper analysis, and nuanced understanding - traits often overlooked by standard SFT. To validate the effectiveness of CFT, we construct multiple critique datasets (e.g., WebInstruct, MetaMath, NuminaMath), where GPT-4o serves as the teacher to generate critiques in the form of ([query; noisy response], critique). Experiments on these datasets demonstrate that CFT consistently outperforms SFT by 4-10% across six mathematical reasoning benchmarks, and is effective across different base models including Qwen2.5, Qwen2.5-Math, and DeepSeek-Math. Notably, our model Qwen2.5-Math-CFT only requires 1 hour of training on 8 x H100 over the 50K examples, yet matches or outperforms strong competitors like Qwen2.5-Math-Instruct on most benchmarks, which use over 2M samples. Moreover, it matches the performance of SimpleRL, which is a DeepSeek-r1 replication trained with 140 x more compute. Experiments on IF_Eval and MT-Bench further demonstrate that CFT can significantly enhance the model's general generation and instruction-following capabilities, outperforming the Qwen2.5-Math-Instruct by a large margin. Ablation studies show that CFT is robust to noisy response sources and teacher critique models. These findings highlight that CFT offers a more effective alternative to advance the reasoning of language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑