Meta Superintelligence Labs(Meta超智能实验室);ELLIS Institute Tübingen(图宾根ELLIS研究所);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Oxford(牛津大学);University of Cambridge(剑桥大学)
Shashwat Goel, Rishi Hazra, Dulhan Jayalath, Timon Willi, Parag Jain, William F. Shen, Ilias Leontiadis, Francesco Barbieri, Yoram Bachrach, Jonas Geiping, Chenxi Whitehouse
备注:
11 pages in the main paper, total 119 including sample outputs in the Appendix
AI co-scientists are emerging as a tool to assist human researchers in achieving their research goals. A crucial feature of these AI co-scientists is the ability to generate a research plan given a set of aims and constraints. The plan may be used by researchers for brainstorming, or may even be implemented after further refinement. However, language models currently struggle to generate research plans that follow all constraints and implicit requirements. In this work, we study how to leverage the vast corpus of existing research papers to train language models that generate better research plans. We build a scalable, diverse training corpus by automatically extracting research goals and goal-specific grading rubrics from papers across several domains. We then train models for research plan generation via reinforcement learning with self-grading. A frozen copy of the initial policy acts as the grader during training, with the rubrics creating a generator-verifier gap that enables improvements without external human supervision. To validate this approach, we conduct a study with human experts for machine learning research goals, spanning 225 hours. The experts prefer plans generated by our finetuned Qwen3-30B-A3B model over the initial model for 70% of research goals, and approve 84% of the automatically extracted goal-specific grading rubrics. To assess generality, we also extend our approach to research goals from medical papers, and new arXiv preprints, evaluating with a jury of frontier models. Our finetuning yields 12-22% relative improvements and significant cross-domain generalization, proving effective even in problem settings like medical research where execution feedback is infeasible. Together, these findings demonstrate the potential of a scalable, automated training recipe as a step towards improving general AI co-scientists.
RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains
University at Albany(阿尔巴尼大学);Rutgers University(罗格斯大学);Purdue University(普渡大学);Independent Researcher(独立研究员);Emory University(埃默里大学);Georgia Institute of Technology(佐治亚理工学院)
C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
The Graduate University for Advanced Studies (SOKENDAI)(高级研究大学(SOKENDAI));National Institute of Informatics(国家信息研究所);The Asahi Shimbun Company(朝日新闻公司);The University of Tokyo(东京大学)
Stanford University(斯坦福大学);Boston University(波士顿大学);Columbia University(哥伦比亚大学);University of Toronto(多伦多大学);Institut Teknologi Bandung(Bandung 工程技术大学);Monash University Indonesia(墨尔本大学印尼分校);Capital One
EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning
National Engineering Research Center of Software Engineering, Peking University(北京大学软件工程国家工程研究中心);School of Computer Science, Peking University(北京大学计算机学院);Key Laboratory of High Confidence Software Technologies, Ministry of Education(教育部高可信软件技术重点实验室);Peking University Information Technology Institute, Tianjin Binhai(北京大学滨海信息技术研究院);Research Institute, GRGBanking Equipment Co., Ltd.(广电运通金融电子股份有限公司研究院)
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University(北京未来区块链与隐私计算先进创新中心,人工智能学院,北京航空航天大学);Tsinghua University(清华大学)
针对准则奖励中因忽略准则间依赖关系导致的虚假信用传播问题,提出概率图框架Graphical Event Aggregation for Rubric rewards (GEAR),通过建模潜在伯努利事件和软抑制传播实现依赖感知的奖励聚合,在多个基准上提升性能并减少信用泄漏。
AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning
The University of Texas at Dallas(德克萨斯大学达拉斯分校);Adobe Inc.(Adobe公司);Department of Computer Science, University of California, Santa Barbara(加州大学圣芭芭拉分校计算机科学系)
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, San Diego(加州大学圣地亚哥分校);Texas A&M University(德克萨斯A&M大学);Stevens Institute of Technology(史蒂文斯理工学院)
DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research
University of Washington(华盛顿大学);Allen Institute for AI(人工智能研究院);Carnegie Mellon University(卡内基梅隆大学);Massachusetts Institute of Technology(麻省理工学院);Seattle Children's Hospital(西雅图儿童医院);University of California, Berkeley(加州大学伯克利分校)