Current large language models (LLMs) often struggle to produce accurate responses on the first attempt for complex reasoning tasks like code generation. Prior research tackles this challenge by generating multiple candidate solutions and validating them with LLM-generated unit tests. The execution results of unit tests serve as reward signals to identify correct solutions. As LLMs always confidently make mistakes, these unit tests are not reliable, thereby diminishing the quality of reward signals. Motivated by the observation that scaling the number of solutions improves LLM performance, we explore the impact of scaling unit tests to enhance reward signal quality. Our pioneer experiment reveals a positive correlation between the number of unit tests and reward signal quality, with greater benefits observed in more challenging problems. Based on these insights, we propose CodeRM-8B, a lightweight yet effective unit test generator that enables efficient and high-quality unit test scaling. Additionally, we implement a dynamic scaling mechanism that adapts the number of unit tests based on problem difficulty, further improving efficiency. Experimental results show that our approach significantly improves performance across various models on three benchmarks (e.g., with gains of 18.43% for Llama3-8B and 3.42% for GPT-4o-mini on HumanEval Plus).
Inference-Time Scaling for Generalist Reward Modeling
DeepSeek-AI;Dept. of Computer Sci. & Tech., Tsinghua University(计算机科学与技术系,清华大学);Institute for AI Industry Research (AIR), Tsinghua University(人工智能产业研究院(AIR),清华大学)
From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling
Meituan Inc.(美团公司);National Key Laboratory for Multimedia Information Processing, Peking University(北京大学多媒体信息处理国家重点实验室);Pennsylvania State University(宾夕法尼亚大学)
本研究全面分析了Process Reward Models的训练方法、可扩展性与泛化能力,发现平衡模型规模与计算成本至关重要,Monte Carlo Tree Search在资源充足时最优,且在数学数据集上训练的PRMs能跨领域泛化至代码生成任务。
Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
Singapore University of Technology and Design(新加坡科技设计大学);National University of Singapore(国立新加坡大学);The University of Tokyo(东京大学);MiroMind(MiroMind公司);Institute for Infocomm Research, A*Star, Singapore(新加坡*星研究院信息与通信研究所)
AgentV-RL: Scaling Reward Modeling with Agentic Verifier
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Huazhong University of Science and Technology(华中科技大学);The University of Hong Kong(香港大学);ByteDance Seed(字节跳动种子);Institute of Trustworthy Embodied AI, Fudan University(复旦大学可信具身人工智能研究院);Shanghai Key Laboratory of Multimodal Embodied AI(上海多模态具身人工智能重点实验室)
Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric
Qwen Large Model Application Team, Alibaba(阿里巴巴文勤大模型应用团队);Beijing University Of Posts and Telecommunications(北京邮电大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
Unifying Tree Search Algorithm and Reward Design for LLM Reasoning: A Survey
Zhejiang University(浙江大学);University of British Columbia(不列颠哥伦比亚大学);Fudan University(复旦大学);Stony Brook University(石溪大学);The Chinese University of Hong Kong(香港中文大学)
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学);Nanjing University(南京大学);Fudan University(复旦大学);Nanyang Technological University(南洋理工大学)