信号重塑用于弱反馈代理代码修复中的GRPO
Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repair
AI总结：
本文研究了在弱反馈环境下，通过信号重塑提升GRPO在代理代码修复中的性能，通过三种信号重塑方法改进轨迹排名、轨迹内信用分配和组内可比性，实验表明信号重塑显著提高了修复准确率。
摘要：
代码代理强化学习常接收弱反馈：回放时的信号可靠且可执行，但仅捕捉任务成功的必要或表面条件，而非目标语义谓词。在代理编译修复的设定下，我们研究了在如此反馈下标准GRPO的信号重塑。我们的核心观点是，GRPO组内比较仅在三种信号重塑后才有意义：结果奖励恢复语义排名，过程信号局部化轨迹内信用，以及相同提示的回放保持执行可比性。我们通过最小的信号重塑构造来操作这些条件，该构造不改变GRPO的组归一化优势构造：编译和语义分层奖励重塑轨迹排名，步骤级过程分数在组奖励归一化外重塑轨迹内更新强度，失败原因意识的回放治理重塑组内可比性。实验显示明显的端到端收益：完全信号重塑的GRPO将严格编译和语义准确率从基模型的零样本0.385提升至0.535。受控比较进一步解释了这一收益来源：二元奖励移除了编译-only中间层并退化轨迹控制；在分层奖励之上，过程分数加权进一步将准确率从0.48提升至0.53并减少平均评估步骤从23.50到17.02。作为边界比较，特权提示词级蒸馏主要优化局部分布对齐；在长工具使用轨迹中，此信号被非关键词稀释，无法替代结果语义、过程信用或组内可比性。
英文摘要：
Code-agent RL often receives weak feedback: rollout-time signals are reliable and executable, but capture only necessary or surface conditions for task success rather than the target semantic predicate. Using agentic compile-fix as the setting, we study signal reshaping for standard GRPO under such feedback. Our central claim is that GRPO's within-group comparison is meaningful only after three kinds of signals are reshaped: outcome rewards recover semantic ranking, process signals localize intra-trajectory credit, and rollouts from the same prompt remain execution-comparable. We operationalize these conditions with a minimal signal-reshaping construction that leaves GRPO's group-normalized advantage construction unchanged: compile-and-semantic layered rewards reshape trajectory ranking, step-level process scores outside group reward normalization reshape within-trajectory update strength, and failure-cause-aware rollout governance reshapes within-group comparability. Experiments show a clear end-to-end gain: full signal-reshaped GRPO improves strict compile-and-semantic accuracy from the base model's zero-shot $0.385$ to $0.535$. Controlled comparisons further explain the source of this gain: binary rewards remove the compile-only middle tier and degrade trajectory control; on top of layered rewards, process-score weighting further improves accuracy from $0.48$ to $0.53$ and reduces average evaluation steps from $23.50$ to $17.02$. As a boundary comparison, privileged-prompt token-level distillation mainly optimizes local distributional alignment; in long tool-use trajectories, this signal is diluted by non-critical tokens and cannot replace outcome semantics, process credit, or within-group comparability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
