基于错误合成数据的强化学习将大语言模型数学推理效率提升八倍
RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold
AI总结:
该研究针对LLM数学推理中合成数据训练效果不稳定的问题,提出基于逐步骤错误负样本的训练方法,等价于优势加权RL,可消除虚假相关性,将合成数据训练效率提升8倍。
摘要:
在模型生成的合成数据上训练是微调大语言模型(LLM)的一种极具前景的方法,但目前尚不清楚这种方法何时有益、何时会产生负面影响。本文针对数学推理场景开展实证研究以探究这一问题,并进一步构建了对观测结果的概念性理解。首先我们发现,典型方法是用能力较强的模型生成的正确/正向问题-解对来微调模型,这种方法仅能带来有限的性能提升;而从微调后的模型自身采样更多正确解,再用这些自生成数据进行后续微调,能让相同合成问题的训练效率提升一倍。与此同时,在模型生成的正向样本上训练可能会放大各类虚假相关性,导致随着数据量增加,性能呈现停滞甚至反向缩放的趋势。令人意外的是,我们发现若同时利用负向响应——即被最终答案验证器判定为错误的模型生成响应——就能解决上述若干问题。关键在于,这些负向样本的构建必须让训练能够恰当还原负向响应中每一个中间步骤的效用或优势。借助这种逐步骤方案,我们能够获得比仅用正向数据更稳定的性能提升,其效果相当于将合成数据量放大8倍。我们证实,基于逐步骤负向样本的训练有助于消除正向数据中的虚假相关性,且等价于优势加权强化学习(RL),这意味着该方法继承了强化学习相较于单纯模仿正向数据的鲁棒性优势。
英文摘要:
Training on model-generated synthetic data is a promising approach for finetuning LLMs, but it remains unclear when it helps or hurts. In this paper, we investigate this question for math reasoning via an empirical study, followed by building a conceptual understanding of our observations. First, we find that while the typical approach of finetuning a model on synthetic correct or positive problem-solution pairs generated by capable models offers modest performance gains, sampling more correct solutions from the finetuned learner itself followed by subsequent fine-tuning on this self-generated data $\textbf{doubles}$ the efficiency of the same synthetic problems. At the same time, training on model-generated positives can amplify various spurious correlations, resulting in flat or even inverse scaling trends as the amount of data increases. Surprisingly, we find that several of these issues can be addressed if we also utilize negative responses, i.e., model-generated responses that are deemed incorrect by a final answer verifier. Crucially, these negatives must be constructed such that the training can appropriately recover the utility or advantage of each intermediate step in the negative response. With this per-step scheme, we are able to attain consistent gains over only positive data, attaining performance similar to amplifying the amount of synthetic data by $\mathbf{8 \times}$. We show that training on per-step negatives can help to unlearn spurious correlations in the positive data, and is equivalent to advantage-weighted reinforcement learning (RL), implying that it inherits robustness benefits of RL over imitating positive data alone.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。