作为智能体自我修正反馈的反例
Counterexamples as Feedback for Agent Self-Correction
浏览论文内容
中文总结 AI 辅助
该研究提出轻量级框架A-CEGIS，以反例为反馈评估自然语言转正则表达式合成的多轮优化，在30项任务中表现优于基线方法，可衡量智能体优化效率并提升稳健性。
中文摘要 AI 辅助
单轮代码生成指标低估了已部署智能体的核心特性：在收到具体反馈后修复错误产物的能力。本文提出A-CEGIS，一个轻量级框架，使用反例作为反馈来评估自然语言转正则表达式（regex）合成中的多轮优化过程。智能体提出正则表达式后，确定性预言机在全匹配语义下对其进行检查，而紧凑的假正例或假负例见证则指导下一轮优化。在30项NL-RX-Turk任务中，诊断性反例反馈在四轮消融预算内解决了90%的任务，而零样本生成、通用自我修正、仅错误反馈的解决率分别为17%、27%、23%。在带强化的完整诊断运行中，最终轮次解决了隐藏集的所有任务，平均成功轮次为2.7，经针对性探测后的稳健成功率为77%。这些结果表明，A-CEGIS可衡量智能体跨轮优化的效率，同时在原保留案例之外增加了实用的稳健性检查。
英文摘要
Single-turn code-generation metrics understate a central property of deployed agents: whether they can repair a wrong artifact after receiving concrete feedback. This paper presents A-CEGIS, a lightweight framework that uses counterexamples as feedback for evaluating multi-turn refinement in natural-language-to-regex synthesis. An agent proposes a regex, a deterministic oracle checks it under full-match semantics, and compact false-positive or false-negative witnesses guide the next turn. On 30 NL-RX-Turk tasks, diagnostic counterexample feedback solves 90\% of tasks within a four-turn ablation budget, compared with 17% for zero-shot generation, 27% for generic self-correction, and 23% for error-only feedback. In a full diagnostic run with hardening, all tasks are solved on the hidden set by the final turn, with mean time-to-success of 2.7 turns and robust success of 77% after targeted probing. These results show that A-CEGIS measures how efficiently an agent improves across turns while adding a practical robustness check beyond the original held-out cases.
