自动化自我测试作为质量门：基于证据的LLM应用发布管理
Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications
AI总结：
本文提出了一种自动化自我测试框架，通过五个实证基础的维度（任务成功率、研究环境保持、P95延迟、安全通过率和证据覆盖）实现基于证据的发布决策（PROMOTE/HOLD/ROLLBACK），并通过长期案例研究评估了该框架在多代理对话AI系统中的有效性。
摘要：
LLM应用是AI系统，其非确定性输出和不断变化的模型行为使得传统测试不足以满足发布管理的需求。我们提出了一种自动化自我测试框架，引入了基于证据的发布决策质量门（PROMOTE/HOLD/ROLLBACK）五个实证基础的维度：任务成功率、研究环境保持、P95延迟、安全通过率和证据覆盖。我们通过一个长期案例研究评估该框架，该研究涉及一个内部部署的多代理对话AI系统，具有特定的营销能力，并在活跃开发中覆盖了38次评估运行，跨越20多个内部发布。质量门在早期运行中识别出两个ROLLBACK级构建，并在四周的 staging 生命周期中支持稳定的质量演变，同时执行了基于角色的、多轮的、对抗性和证据要求的场景。统计分析（Mann-Kendall趋势、Spearman相关性、bootstrap置信区间）、质量门消融和开销扩展表明，证据覆盖是主要的严重回归判别器，且运行时间与套件大小成比例增长。人类校准研究（n=60分层案例，两个独立评估者，LLM-as-judge交叉验证）揭示了互补的多模态覆盖：LLM-judge与系统门的分歧（kappa=0.13）可归因于结构失败模式——延迟违规和路由错误——这些在响应文本中是不可见的，而评估者独立地揭示了被结构检查遗漏的内容质量失败，这与多维门设计一致。该框架、补充伪代码和校准工件被提供以支持AI系统质量保证和独立复制。
英文摘要：
LLM applications are AI systems whose nondeterministic outputs and evolving model behavior make traditional testing insufficient for release governance. We present an automated self-testing framework that introduces quality gates with evidence-based release decisions (PROMOTE/HOLD/ROLLBACK) across five empirically grounded dimensions: task success rate, research context preservation, P95 latency, safety pass rate, and evidence coverage. We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system with specific marketing capabilities in active development, covering 38 evaluation runs across 20+ internal releases. The gate identified two ROLLBACK-grade builds in early runs and supported stable quality evolution over a four-week staging lifecycle while exercising persona-grounded, multi-turn, adversarial, and evidence-required scenarios. Statistical analysis (Mann-Kendall trends, Spearman correlations, bootstrap confidence intervals), gate ablation, and overhead scaling indicate that evidence coverage is the primary severe-regression discriminator and that runtime scales predictably with suite size. A human calibration study (n=60 stratified cases, two independent evaluators, LLM-as-judge cross-validation) reveals complementary multi-modal coverage: LLM-judge disagreements with the system gate (kappa=0.13) are attributable to structural failure modes - latency violations and routing errors - invisible in response text alone, while the judge independently surfaces content quality failures missed by structural checks, consistent with a multi-dimensional gate design. The framework, supplementary pseudocode, and calibration artifacts are provided to support AI-system quality assurance and independent replication.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
