RECAP: 提示持续适应的回归评估
RECAP: Regression Evaluation for Continual Adaptation of Prompts
- Capital One
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
提出RECAP基准，在严格主动适应-测试协议下评估提示优化方法对约束变化的持续学习能力，发现现有方法在主动场景下性能无显著提升，强调设计主动提示适应方法的必要性。
AI中文摘要：
生产中的代理系统经常面临不断变化的约束，并且必须从下一次交互开始就遵守。诸如工具调用通知更改合规阈值或策略更新添加披露要求等场景符合这一标准，在生产中几乎没有出错的空间。这种主动适应设置在部署中很常见，但在当前的基准测试中却不存在，这些基准测试假设要么是静态约束集，要么是带有评估反馈的反应式协议。我们引入了RECAP，这是一个基准测试，在严格主动适应-测试协议下，在约束级别测量持续学习现象（遗忘、回归、前向转移）：提示优化方法仅接收约束规范，并且必须在看到任何测试数据之前进行泛化。我们在四个LLM和三个具有不断变化的约束的调度上评估了六种方法，发现这些方法在性能上没有显著改善，即使在产生更高延迟之后也是如此。这些为离线或反应式设置设计的方法不足以应对主动范式。我们的工作强调了设计主动提示适应方法的日益增长的需求，其中模型必须对部署中不断变化的需求保持鲁棒性。
英文摘要：
Production agentic systems routinely face evolving constraints and must comply from the very next interaction. Scenarios like a tool-call notification changing a compliance threshold or a policy update adding disclosure requirements fit this criteria, having close to no room for errors in production. This proactive adaptation setting is common in deployment, but absent from current benchmarks, which assume either static constraint sets or reactive protocols with evaluation feedback. We introduce RECAP, a benchmark that measures continual-learning phenomena (forgetting, regression, forward transfer) at the constraint level under a strictly proactive adapt-then-test protocol: prompt optimization methods receive only the constraint specification and must generalize before seeing any test data. Evaluating six methods across five LLMs and three schedules with evolving constraints, we find that these methods show no significant improvement in performance, even after incurring a higher latency. These methods, designed for offline or reactive settings, are inadequate for the proactive paradigm. Our work emphasizes the growing need for designing proactive prompt adaptation methods, where the models must remain robust to evolving needs in deployment.
