arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.23707cs.LGcs.CLcs.HC
收藏

利用评分奖励训练AI合作者

Training AI Co-Scientists Using Rubric Rewards

Meta Superintelligence Labs(Meta超智能实验室);ELLIS Institute Tübingen(图宾根ELLIS研究所);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Oxford(牛津大学);University of Cambridge(剑桥大学)

Shashwat Goel, Rishi Hazra, Dulhan Jayalath, Timon Willi, Parag Jain, William F. Shen, Ilias Leontiadis, Francesco Barbieri, Yoram Bachrach, Jonas Geiping, Chenxi Whitehouse

AI总结:

本文提出利用评分奖励训练AI合作者,通过自动生成研究计划并改进模型性能,实现跨领域泛化和高效训练。

摘要:

AI合作者正逐渐成为帮助人类研究人员实现研究目标的工具。这些AI合作者的一个关键特征是能够在给定一组目标和约束的情况下生成研究计划。该计划可以用于研究人员的头脑风暴,甚至在进一步完善后被实施。然而,目前的语言模型在生成遵循所有约束和隐含要求的研究计划方面存在困难。在本工作中,我们研究如何利用现有研究论文的庞大语料库来训练能够生成更好研究计划的语言模型。我们通过自动从多个领域论文中提取研究目标和目标特定的评分标准,构建了一个可扩展且多样化的训练语料库。然后,我们通过自我评分的强化学习训练模型进行研究计划生成。初始策略的冻结副本在训练过程中充当评分者,评分标准创造了生成者-验证者之间的差距,从而在无需外部人类监督的情况下实现改进。为了验证这种方法,我们与人类专家进行了一项研究,涵盖机器学习研究目标,持续了225小时。专家们在70%的研究目标中更喜欢我们微调的Qwen3-30B-A3B模型生成的计划,且批准了84%的自动提取的目标特定评分标准。为了评估通用性,我们还将这种方法扩展到医学论文和新的arXiv预印本,通过前沿模型的评审团进行评估。我们的微调带来了12-22%的相对改进和显著的跨领域泛化能力,证明了在医疗研究等执行反馈不可行的问题设置中也有效。这些发现共同展示了可扩展、自动化的训练配方在改进通用AI合作者方面的潜力。

英文摘要:

AI co-scientists are emerging as a tool to assist human researchers in achieving their research goals. A crucial feature of these AI co-scientists is the ability to generate a research plan given a set of aims and constraints. The plan may be used by researchers for brainstorming, or may even be implemented after further refinement. However, language models currently struggle to generate research plans that follow all constraints and implicit requirements. In this work, we study how to leverage the vast corpus of existing research papers to train language models that generate better research plans. We build a scalable, diverse training corpus by automatically extracting research goals and goal-specific grading rubrics from papers across several domains. We then train models for research plan generation via reinforcement learning with self-grading. A frozen copy of the initial policy acts as the grader during training, with the rubrics creating a generator-verifier gap that enables improvements without external human supervision. To validate this approach, we conduct a study with human experts for machine learning research goals, spanning 225 hours. The experts prefer plans generated by our finetuned Qwen3-30B-A3B model over the initial model for 70% of research goals, and approve 84% of the automatically extracted goal-specific grading rubrics. To assess generality, we also extend our approach to research goals from medical papers, and new arXiv preprints, evaluating with a jury of frontier models. Our finetuning yields 12-22% relative improvements and significant cross-domain generalization, proving effective even in problem settings like medical research where execution feedback is infeasible. Together, these findings demonstrate the potential of a scalable, automated training recipe as a step towards improving general AI co-scientists.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑