迈向数学错误纠正中的人工智能导师的奖励建模
Towards Reward Modeling for AI Tutors in Math Mistake Remediation
AI总结:
本文提出了一种基于人类偏好的人工智能导师奖励建模方法,通过合成数据和加权和数据提升模型准确性,达到0.74的配对准确率。
摘要:
评估人工智能导师的教学质量仍然具有挑战性:标准的自然语言生成(NLG)度量标准无法确定响应是否识别错误、构建推理、或避免透露答案。在错误纠正任务中,我们从MRBench上的人类成对偏好中推导出一个教学方面的层次结构,并合成最小对比响应对,这些响应在关键方面(如错误识别和定位、针对性、支架作用、可操作性、清晰性和连贯性)上有所不同。我们开发并发布了基于加权和排名的布雷德利-蒂尔利偏好模型,这些模型训练于从MRBench、合成对和数据组合中自动创建的加权和排名。仅使用合成数据,我们的最佳模型在人类偏好测试中达到0.69的配对准确率,结合加权和数据与针对性合成组可以将准确率提高到0.74,优于更大的通用奖励模型,同时仅使用0.5B参数的模型架构。
英文摘要:
Evaluating the pedagogical quality of AI tutors remains challenging: standard NLG metrics do not determine whether responses identify mistakes, scaffold reasoning, or avoid revealing the answers. For the task of mistake remediation, we derive a hierarchy of pedagogical aspects from human pairwise preferences on MRBench, and synthesize minimally contrastive response pairs that differ along key aspects (e.g., mistake identification and location, targetedness, scaffolding, actionability, clarity, and coherence). We develop and release Bradley-Terry preference models trained on weighted-sum rankings that we automatically create from MRBench, synthetic pairs, and data combinations. Using only synthetic data, our best model reaches 0.69 pairwise accuracy on a human preference test, and combining weighted-sum data with targeted synthetic groups improves accuracy to 0.74, outperforming larger general-purpose reward models while using only a 0.5B-parameter backbone.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。