可复现评估作为生产智能体中安全提示压缩的基础
Reproducible Evaluation as the Substrate for Safe Prompt Compression in Production Agents
AI总结:
该研究提出面向生产对话智能体的可复现离线评估工具,将其作为带正确性门控的GEPA提示优化的适应度函数,实现了面向商家的智能体提示压缩且未降低任务正确性。
摘要:
评估作用于实时可变企业数据的对话智能体颇具挑战:随着后端状态变化,重放会话会出现偏差,且今日构建的基准会因所描述数据的更新而悄然失效。我们提出一种面向生产对话智能体的离线评估工具,尽管其运行的实时数据发生变化,该工具仍能保持可复现性,且一旦失效可按需重新生成。随后,我们将此工具作为基于GEPA的提示优化的适应度函数,该优化受正确性门控约束,以防范奖励黑客行为。将其应用于面向商家的生产智能体,经正确性门控的压缩可使智能体的完整提示表面缩减3.5%-7.6%,且在三个独立的主体模型系列中均未出现保留任务正确性的退化,其中一个系列的正确性还可得到显著提升。
英文摘要:
Evaluating conversational agents that act on live, mutable enterprise data is hard: replayed sessions drift as backend state changes, and a benchmark built today can silently go stale as the data it describes moves on. We present an offline evaluation harness for a production conversational agent that stays reproducible despite the live data it operates on changing, and that can be regenerated on demand once it goes stale. We then use this harness as the fitness function for GEPA-based prompt optimization under a correctness gate that guards against reward hacking. Applied to a production merchant-facing agent, correctness-gated compression shrinks the agent's full prompt surface by 3.5-7.6% without regressing held-out task correctness, consistently across three independent subject-model families, and for one, correctness measurably improves.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。