Inquesto Score：语音代理的可靠性协议

中文摘要 AI 辅助

语音代理越来越多地部署在工作流程中，在这些流程中，失败的交互可能影响交易、访问和其他具有重大后果的结果，从而产生了对可复现且可解释的评估的需求。我们引入了Inquesto Score（IS），这是一种衡量语音代理可靠性的协议，其定义为在固定的、版本化的评估总体中，达到呼叫者目标且未发生功能性故障或更糟情况的呼叫百分比。IS不是组合异构指标，而是定义明确的故障事件和严重性级别，并评估已部署的语音流水线。时序故障，包括重叠讲话和延迟响应，直接从音频中测量，而语义和状态依赖的故障则使用场景谓词、工具轨迹和固定的开放模型评判器进行评估。行为、声学鲁棒性、身份处理和说话者群体的诊断视图伴随评分，但不合并到评分中。Inquesto Score v0.1评估了30个场景、三种声学条件、四个说话者群体，以及参考语音代理系统的13种配置下每个代理的306次呼叫。我们的评估表明，可靠的测量需要超越转录的证据、对部署条件的明确处理，以及用于确定结果的评估器的验证。我们发布了协议、参考实现和评估记录。

英文摘要

Voice agents are increasingly deployed in workflows where failed interactions can affect transactions, access, and other consequential outcomes, creating a need for reproducible and interpretable evaluation. We introduce Inquesto Score (IS), a protocol for measuring voice-agent reliability as the percentage of calls in a fixed, versioned evaluation population that achieve the caller's goal without a functional failure or worse. Rather than combining heterogeneous metrics, IS defines explicit failure events and severity levels and evaluates the deployed voice pipeline. Timing failures, including talk-over and delayed responses, are measured directly from audio, while semantic and state-dependent failures are evaluated using scenario predicates, tool traces, and a pinned open-model judge. Diagnostic views of behavior, acoustic robustness, identity handling, and speaker groups accompany the score without being combined into it. Inquesto Score v0.1 evaluates 30 scenarios, three acoustic conditions, four speaker groups, and 306 calls per agent across 13 configurations of a reference voice-agent system. Our evaluation shows that reliable measurement requires evidence beyond transcripts, explicit treatment of deployment conditions, and validation of the evaluators used to determine outcomes. We release the protocol, reference implementation, and evaluation records.

发表机构

• Carnegie Mellon University（卡内基梅隆大学）

机构由 AI 辅助整理，请以论文原文为准。