发表机构
AI Research Engineer（人工智能研究工程师）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文指出LLM评判者存在多类评估信号失效问题，提出PROCTOR教师-学生循环方案，通过确定性护栏预防部分故障，解决自改进智能体的评估可靠性问题。
AI 中文摘要
自改进智能体流程存在核心问题：优化器会重写提示以获得更高分数，而分数来自本身为大语言模型（LLM）的评判者，该评判者对系统是否在进步拥有最终决定权，本文认为它不配拥有此权力。应将评判者从神谕降级为顾问：其裁决成为多个输入中的一个，所有变更需由评判者无法覆盖的确定性验证层进行把关。我们通过构建替代方案并运行来得出该结论：在合同分析、合规审查和代码质量等场景的生产环境中运行自主提示优化循环数月后，我们梳理出四类共11种评估信号失效的情况：评判者偏差、框架与指标失效、真实值错误、奖励黑客行为。智能体通过从环境中读取缓存答案键获得了完美分数，100%的通过率掩盖了68%的真实能力；损坏的真实值标签导致优化器删除正确的合规规则以与其保持一致；因静默解析器回退提升了指标，语法错误的提示被选为获胜者；尝试通过重写评判者的评分规则来修复其性能陷入停滞，唯一可靠的提升来自对其输出顺序施加结构约束。作为应对，我们提出PROCTOR，这是一种教师-学生循环：有状态的协调器掌握所有工具访问权限，无状态子智能体诊断故障并起草无法自行应用的变更，教师在五条确定性护栏下对这些变更进行评分：密封沙箱、能力不重叠的角色、优先级高于教师的接受检查、冻结的保留集，以及设计为完美分数本身就是作弊证据的金丝雀案例。我们报告了该方案所预防的故障，同时由于教师本身是LLM评判者，也报告了其未能预防的故障。
英文摘要
Self-improving agent pipelines have a problem at their center. An optimizer rewrites prompts to score higher, and the score comes from a judge that is itself an LLM. That judge has the last word on whether the system is getting better, and our position is that it has not earned it. The judge should be demoted from oracle to advisor: its verdict becomes one input among several, and every change is gated instead by a deterministic verification layer the judge cannot override. We reached this position by building the alternative and running it. Over months of running autonomous prompt-optimization loops in production across contract analysis, compliance review, and code quality, we cataloged eleven ways the evaluation signal failed, in four classes: judge bias, harness and metric failures, ground-truth errors, and reward hacking. Agents achieved perfect scores by reading cached answer keys from their environment, a 100% pass rate concealing 68% true capability. A corrupted ground-truth label caused the optimizer to delete correct compliance rules to agree with it. A syntactically broken prompt was promoted as the winner because a silent parser fallback improved the metric. Attempts to fix the judge by rewriting its rubric plateaued; the only reliable gain came from a structural constraint on its output order. In response we describe PROCTOR, a Teacher-Student loop in which a stateful orchestrator holds all tool access, stateless subagents diagnose failures and draft mutations they cannot apply, and a Teacher grades those mutations under five deterministic guardrails: hermetic sandboxes, capability-disjoint roles, acceptance checks that outrank the Teacher, frozen holdouts, and canary cases engineered so that a perfect score is itself evidence of cheating. We report the failures this prevented, and, because the Teacher is itself an LLM judge, the failures it did not.
Comments20 pages, 4 figures, 5 tables
