CodeRL+: 通过执行语义对齐改进代码生成
CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment
AI总结：
本文提出CodeRL+，通过将执行语义对齐整合到RLVR训练流程中，提升代码生成的准确性，实验显示其在pass@1指标上平均提升4.6%。
摘要：
尽管大型语言模型（LLMs）通过学习庞大的代码语料库在代码生成方面表现出色，但其训练基于文本模式与功能性正确性目标之间仍存在根本性的语义鸿沟，这一鸿沟由形式执行语义所决定。Reinforcement Learning with Verifiable Rewards（RLVR）方法尝试通过执行测试用例的结果奖励来弥合这一鸿沟。然而，仅依赖二元通过/失败信号在建立代码文本表示与执行语义之间的良好对齐连接方面效率低下，特别是对于代码中的细微逻辑错误。在本文中，我们提出CodeRL+，一种新的方法，将执行语义对齐整合到代码生成的RLVR训练流程中。CodeRL+使模型能够推断变量级的执行轨迹，提供执行语义的直接学习信号。CodeRL+能够利用现有的on-policy回滚直接构建执行语义对齐，并与各种RL算法无缝集成。广泛的实验表明，CodeRL+优于后训练基线（包括RLVR和Distillation），在pass@1指标上平均提升4.6%。CodeRL+在其他编码任务中表现出良好的泛化能力，在代码推理和测试输出生成基准上分别获得15.5%和4.4%的更高准确性。CodeRL+在各种RL算法和LLM上表现出强大的适用性。此外，探针分析提供了有力的证据，表明CodeRL+增强了代码文本表示与其底层执行语义之间的对齐。
英文摘要：
While Large Language Models (LLMs) excel at code generation by learning from vast code corpora, a fundamental semantic gap remains between their training on textual patterns and the goal of functional correctness, which is governed by formal execution semantics. Reinforcement Learning with Verifiable Rewards (RLVR) approaches attempt to bridge this gap using outcome rewards from executing test cases. However, solely relying on binary pass/fail signals is inefficient for establishing a well-aligned connection between the textual representation of code and its execution semantics, especially for subtle logical errors within the code. In this paper, we propose CodeRL+, a novel approach that integrates execution semantics alignment into the RLVR training pipeline for code generation. CodeRL+ enables the model to infer variable-level execution trajectory, providing a direct learning signal of execution semantics. CodeRL+ can construct execution semantics alignment directly using existing on-policy rollouts and integrates seamlessly with various RL algorithms. Extensive experiments demonstrate that CodeRL+ outperforms post-training baselines (including RLVR and Distillation), achieving a 4.6% average relative improvement in pass@1. CodeRL+ generalizes effectively to other coding tasks, yielding 15.5% and 4.4% higher accuracy on code-reasoning and test-output-generation benchmarks, respectively. CodeRL+ shows strong applicability across diverse RL algorithms and LLMs. Furthermore, probe analyses provide compelling evidence that CodeRL+ strengthens the alignment between code's textual representations and its underlying execution semantics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
