从编译器和语言服务器反馈中学习强化学习
Reinforcement Learning from Compiler and Language Server Feedback
AI总结:
本文提出RLCSF方法和Lanser-CLI工具,通过编译器和语言服务器的反馈信号提升编码代理的可靠性,利用确定性奖励函数和分析捆绑技术实现过程监督。
摘要:
编码代理在文本层面的猜测超过程序事实时会失败:它们会编造API、漂移到错误的符号,并在没有证据的情况下应用编辑。编译器、类型检查器和语言服务器已经计算了缺失的监督信号,但通过为人类驱动的IDE设计的接口暴露出来,而不是学习循环。本文引入了从编译器和语言服务器反馈中学习强化学习(RLCSF)以及Lanser-CLI,一个以命令行接口优先的编排层,将此信号暴露给代理和持续集成。RLCSF将每个工具交互视为一个转移,并从诊断、选择器信心和编辑安全性的确定性变化中计算形状过程奖励。Lanser-CLI将临时LSP会话转换为可回放的分析捆绑,带有固定环境元数据和稳定的内内容哈希。其核心机制包括超越文件:行:列的稳健选择器、确定性捆绑标准化、先查看的受保护突变以及一个奖励函数,其潜在组件在冻结快照下可回放。我们正式化了标准捆绑的确定性,并证明在未折扣设置下,组件逐步改进的转移会获得非负奖励。这些组成部分共同构成了一个实用的编码代理过程监督的基础。
英文摘要:
Coding agents fail when text-level guesses outrun program facts: they hallucinate APIs, drift to the wrong symbol, and apply edits without evidence that the workspace remains valid. Compilers, type checkers, and language servers already compute the missing supervision signal, in the form of diagnostics, symbol resolution, type information, references, and refactoring preconditions, but expose it through interfaces designed for human-driven IDEs rather than learning loops. We introduce Reinforcement Learning from Compiler and Language Server Feedback (RLCSF) together with Lanser-CLI, a CLI-first orchestration layer that exposes this signal to agents and CI. RLCSF treats each tool interaction as a transition and computes a shaped process reward from deterministic changes in diagnostics, selector confidence, and edit safety. Lanser-CLI, in turn, converts ephemeral LSP sessions into replayable Analysis Bundles with pinned environment metadata and stable content hashes. Its core mechanisms are robust selectors that go beyond file:line:col, deterministic bundle normalization, preview-first guarded mutations, and a reward functional whose potential-based component is replayable under frozen snapshots. We formalize determinism for canonical bundles and prove that componentwise-improving transitions receive non-negative reward in the undiscounted setting. Together, these pieces yield a practical substrate for process supervision of coding agents.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。