arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.06795cs.CLcs.AI
收藏

生成数据驱动的推理评分标准以实现领域自适应奖励建模

Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modeling

Johns Hopkins University(约翰霍普金斯大学);Amazon Web Services(亚马逊网络服务)

Kate Sanders, Nathaniel Weir, Sapana Chaudhary, Kaj Bostrom, Huzefa Rangwala

AI总结:

本文提出数据驱动的推理评分标准,用于提升领域自适应奖励建模的错误检测能力,实验显示其在技术领域任务准确性提升达45%

摘要:

使用大型语言模型(LLMs)进行推理输出验证的障碍在于LLMs难以可靠地识别推理轨迹中的错误,尤其是在长输出、需要专业知识的领域以及没有可验证奖励的问题中。我们提出了一种数据驱动的方法,自动构建高度细粒度的推理错误分类学,以增强LLM驱动的错误检测能力,以应对未见过的推理轨迹。我们的发现表明,在技术领域如编程、数学和化学工程中,利用这些错误分类学或“评分标准”的分类方法,在错误识别方面比基线方法表现更强。这些评分标准可以用来构建更强大的LLM-as-judge奖励函数,以通过强化学习训练推理模型。实验结果表明,这些奖励有潜力在困难领域中将模型的任务准确性提高45%以上,而使用仅20%的黄金标签即可达到使用可验证奖励训练的模型性能。通过我们的方法,我们将奖励评分标准的应用从评估模型行为的定性方面扩展到评估任务中通过RLVR奖励通常学习的定量模型正确性。这种扩展为无需完整黄金标签数据集的教学模型解决复杂技术问题打开了大门,而这些数据集通常成本很高。

英文摘要:

An impediment to using Large Language Models (LLMs) for reasoning output verification is that LLMs struggle to reliably identify errors in thinking traces, particularly in long outputs, domains requiring expert knowledge, and problems without verifiable rewards. We propose a data-driven approach to automatically construct highly granular reasoning error taxonomies to enhance LLM-driven error detection on unseen reasoning traces. Our findings indicate that classification approaches that leverage these error taxonomies, or "rubrics", demonstrate strong error identification compared to baseline methods in technical domains like coding, math, and chemical engineering. These rubrics can be used to build stronger LLM-as-judge reward functions for reasoning model training via reinforcement learning. Experimental results show that these rewards have the potential to improve models' task accuracy on difficult domains over models trained by general LLMs-as-judges by +45%, and approach performance of models trained by verifiable rewards while using as little as 20% as many gold labels. Through our approach, we extend the usage of reward rubrics from assessing qualitative model behavior to assessing quantitative model correctness on tasks typically learned via RLVR rewards. This extension opens the door for teaching models to solve complex technical problems without a full dataset of gold labels, which are often highly costly to procure.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑