arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.11226cs.SE
收藏

ExecVerify: 带可验证逐步奖励的白盒强化学习用于代码执行推理

ExecVerify: White-Box RL with Verifiable Stepwise Rewards for Code Execution Reasoning

The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全全国重点实验室);University College London(伦敦大学学院)

Lingxiao Tang, He Ye, Zhaoyang Chu, Muyang Ye, Zhongxin Liu, Xiaoxue Ren, Lingfeng Bao

AI总结:

ExecVerify通过可验证的白盒奖励提升代码执行推理和生成性能

摘要:

代码大语言模型仍难以处理代码执行推理,尤其是在小模型中。现有方法依赖于监督微调(SFT)与教师生成的解释,主要以两种形式存在:(1)输入输出(I/O)预测链和(2)执行轨迹的自然语言描述。然而,在SFT过程中无法显式验证中间执行步骤,因此训练目标只能减少到匹配教师解释。此外,训练数据通常是未经显式控制任务难度而收集的。我们引入ExecVerify,超越文本模仿,通过结合可验证的白盒奖励,这些奖励来自执行轨迹,包括下一步语句预测和变量值/类型预测。我们的工作首先通过约束基础程序合成构建了一个具有多个难度级别的数据集。然后,我们应用强化学习(RL)来奖励关于中间执行步骤和最终输出的正确答案,使训练目标与每一步的语义正确性对齐。最后,我们采用两阶段训练流程,首先增强执行推理,然后转移到代码生成。实验表明,一个7B模型在使用ExecVerify训练后,在代码推理基准上表现与32B模型相当,并在代码生成任务上比强大的后训练基线提高了高达5.9%的pass@1。

英文摘要:

Code LLMs still struggle with code execution reasoning, especially in smaller models. Existing methods rely on supervised fine-tuning (SFT) with teacher-generated explanations, primarily in two forms: (1) input-output (I/O) prediction chains and (2) natural-language descriptions of execution traces. However, intermediate execution steps cannot be explicitly verified during SFT, so the training objective can reduce to merely matching teacher explanations. Moreover, training data is typically collected without explicit control over task difficulty. We introduce ExecVerify, which goes beyond text imitation by incorporating verifiable white-box rewards derived from execution traces, including next-statement prediction and variable value/type prediction. Our work first builds a dataset with multiple difficulty levels via constraint-based program synthesis. Then, we apply reinforcement learning (RL) to reward correct answers about both intermediate execution steps and final outputs, aligning the training objective with semantic correctness at each execution step. Finally, we adopt a two-stage training pipeline that first enhances execution reasoning and then transfers to code generation. Experiments demonstrate that a 7B model trained with ExecVerify achieves performance comparable to 32B models on code reasoning benchmarks and improves pass@1 by up to 5.9\% on code generation tasks over strong post-training baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑