arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.01054cs.CLcs.SE
收藏

面向代码奖励建模的单元测试动态扩展

Dynamic Scaling of Unit Tests for Code Reward Modeling

Renmin University of China(中国人民大学);Tsinghua University(清华大学);Zhipu AI(智谱AI)

Zeyao Ma, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, Jie Tang

AI总结:

本文针对LLM代码生成中单元测试奖励信号不可靠的问题,提出轻量级单元测试生成器CodeRM-8B和基于问题难度的动态扩展机制,在多个基准上显著提升模型性能。

摘要:

当前的大语言模型(LLMs)在代码生成等复杂推理任务中,往往难以在首次尝试时生成准确的响应。先前的研究通过生成多个候选解决方案并使用LLM生成的单元测试进行验证来应对这一挑战。单元测试的执行结果被用作奖励信号,以识别正确的解决方案。由于LLM总是自信地犯错,这些单元测试并不可靠,从而降低了奖励信号的质量。受扩展解决方案数量能够提升LLM性能这一观察的启发,我们探索了扩展单元测试数量以增强奖励信号质量的影响。我们的先导实验揭示了单元测试数量与奖励信号质量之间存在正相关关系,且在更具挑战性的问题中观察到更大的收益。基于这些见解,我们提出了CodeRM-8B,一个轻量级但有效的单元测试生成器,能够实现高效且高质量的单元测试扩展。此外,我们实现了一种动态扩展机制,根据问题难度自适应地调整单元测试数量,从而进一步提高效率。实验结果表明,我们的方法在三个基准测试上显著提升了多种模型的性能(例如,在HumanEval Plus上,Llama3-8B提升了18.43%,GPT-4o-mini提升了3.42%)。

英文摘要:

Current large language models (LLMs) often struggle to produce accurate responses on the first attempt for complex reasoning tasks like code generation. Prior research tackles this challenge by generating multiple candidate solutions and validating them with LLM-generated unit tests. The execution results of unit tests serve as reward signals to identify correct solutions. As LLMs always confidently make mistakes, these unit tests are not reliable, thereby diminishing the quality of reward signals. Motivated by the observation that scaling the number of solutions improves LLM performance, we explore the impact of scaling unit tests to enhance reward signal quality. Our pioneer experiment reveals a positive correlation between the number of unit tests and reward signal quality, with greater benefits observed in more challenging problems. Based on these insights, we propose CodeRM-8B, a lightweight yet effective unit test generator that enables efficient and high-quality unit test scaling. Additionally, we implement a dynamic scaling mechanism that adapts the number of unit tests based on problem difficulty, further improving efficiency. Experimental results show that our approach significantly improves performance across various models on three benchmarks (e.g., with gains of 18.43% for Llama3-8B and 3.42% for GPT-4o-mini on HumanEval Plus).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑