LLM Readiness Harness: 评估、可观测性和持续集成门禁用于LLM/RAG应用
LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications
AI总结:
本文提出了一种LLM和RAG应用的准备性框架,通过自动化基准测试、OpenTelemetry可观测性和持续集成质量门禁,将评估转化为部署决策流程,并通过帕累托前沿计算场景加权的准备度分数,展示了在票务路由工作流和BEIR接地任务上的评估结果。
摘要:
我们提出了一种用于LLM和RAG应用的准备性框架,将评估转化为部署决策流程。该系统结合了自动化基准测试、OpenTelemetry可观测性和持续集成质量门禁,通过最小的API合同聚合工作流程成功、政策合规性、 groundedness、检索命中率、成本和p95延迟,计算出场景加权的准备度分数。我们对票务路由工作流和BEIR接地任务(SciFact和FiQA)进行了评估,覆盖了完整的Azure矩阵(162/162有效单元跨数据集、场景、检索深度、种子和模型)。结果表明,准备度不是单一指标:在FiQA中,sla-first at k=5时,gpt-4.1-mini在准备度和忠实度上领先,而gpt-5.2则支付了显著的延迟成本;在SciFact中,模型质量接近但仍有操作区分。票务路由回归门禁持续拒绝不安全的提示变体,证明了该框架能够阻止风险发布,而不仅仅是报告离线分数。结果是一个可重复、操作基础的框架,用于决定LLM或RAG系统是否准备好发布。
英文摘要:
We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow. The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract, then aggregates workflow success, policy compliance, groundedness, retrieval hit rate, cost, and p95 latency into scenario-weighted readiness scores with Pareto frontiers. We evaluate the harness on ticket-routing workflows and BEIR grounding tasks (SciFact and FiQA) with full Azure matrix coverage (162/162 valid cells across datasets, scenarios, retrieval depths, seeds, and models). Results show that readiness is not a single metric: on FiQA under sla-first at k=5, gpt-4.1-mini leads in readiness and faithfulness, while gpt-5.2 pays a substantial latency cost; on SciFact, models are closer in quality but still separable operationally. Ticket-routing regression gates consistently reject unsafe prompt variants, demonstrating that the harness can block risky releases instead of merely reporting offline scores. The result is a reproducible, operationally grounded framework for deciding whether an LLM or RAG system is ready to ship.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。