arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.20533cs.LGcs.CL
收藏

穿越复杂性:什么构成了困难数学推理任务的优质监督?

Guiding Through Complexity: What Makes Good Supervision for Hard Math Reasoning Tasks?

Tsinghua University(清华大学);University of California, Los Angeles(加州大学洛杉矶分校)

Xuan He, Da Yin, Nanyun Peng

AI总结:

本文探究弱教师模型如何有效监督LLM完成困难数学推理任务,发现高错误率的困难任务监督优于完美但简单的子任务监督,并揭示步骤级错误率是关键因素,补充子任务监督可进一步提升性能。

摘要:

如何让“弱教师模型”(例如普通人类标注者或现有AI系统)有效监督LLM,以提高其在困难推理任务上的表现,尤其是那些对教师模型构成挑战并需要专业知识或日常练习的任务?在本文中,我们通过研究各种数据驱动策略,在不同复杂度的任务上提供不同质量水平的监督数据,来寻求对这一问题的实证答案。在对齐训练期间,教师模型提供监督的两种直观策略应运而生:1)使用与目标推理任务难度匹配的完整任务中的较低质量监督,2)利用来自难度较低的子任务的更高质量监督。有趣的是,我们发现即使困难任务监督的结果错误率很高(例如90%),在多个困难数学基准上,基于此类数据的训练也能优于对较容易子任务的完美正确监督。我们进一步识别出一个影响训练性能的更关键因素:步骤级错误率,它指示解决方案中错误的严重程度。具体而言,在相同结果错误率但不同步骤级错误率的困难任务监督上训练,在MATH基准上可能导致30%的准确率差距。我们的结果还揭示,用相应的子任务监督补充困难任务监督,比简单地将改写的困难完整任务监督组合起来,能带来显著的性能提升,这为数据增强提供了新途径。数据和代码发布于https://github.com/hexuan21/Weak-to-Strong。

英文摘要:

How can "weak teacher models" such as average human annotators or existing AI systems, effectively supervise LLMs to improve performance on hard reasoning tasks, especially those that challenge and requires expertise or daily practice from the teacher models? In this paper, we seek for empirical answers to this question by investigating various data-driven strategies that offer supervision data at different quality levels upon tasks of varying complexity. Two intuitive strategies emerge for teacher models to provide supervision during alignment training: 1) using lower-quality supervision from complete tasks that match the difficulty of the target reasoning tasks, and 2) leveraging higher-quality supervision from easier subtasks that are less challenging. Interestingly, we find that even when the outcome error rate for hard task supervision is high (e.g., 90\%), training on such data can outperform perfectly correct supervision of easier subtasks on multiple hard math benchmarks. We further identify a more critical factor influencing training performance: step-wise error rates, which indicate the severity of errors in solutions. Specifically, training on hard task supervision with the same outcome error rates but disparate step-wise error rates can lead to a 30\% accuracy gap on MATH benchmark. Our results also reveal that supplementing hard task supervision with the corresponding subtask supervision can yield notable performance improvements than simply combining rephrased hard full task supervision, suggesting new avenues for data augmentation. Data and code are released at https://github.com/hexuan21/Weak-to-Strong.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑