arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.17619cs.CL
收藏

ASR错误何时重要?面向使用工具的语音智能体的任务角色诊断

When Does an ASR Error Matter? Task-Role Diagnostics for Tool-Using Voice Agents

Leibniz University Hannover(汉诺威莱布尼茨大学)

Xiaolong Wang, Shijie Su, Naqi Xiong

AI总结:

该研究针对面向工具使用的语音智能体,提出SpeechEvalX配对诊断协议,发现ASR错误的下游影响取决于受影响的任务角色,数字和否定编辑对工具调用准确率的损失影响远大于控制编辑。

摘要:

在级联式语音智能体中,ASR(自动语音识别)错误会产生截然不同的下游后果:微小的转录错误可能不会改变工具调用,也可能改变其意图或参数。然而,传统的词错误率(WER)无法捕捉这些差异。我们提出SpeechEvalX,一种配对诊断协议,用于评估受ASR错误影响的任务角色如何影响结构化工具调用。受控编辑与源话语、编辑数量及操作相匹配,同时将任务角色标签与词汇严重程度、BERT余弦距离、Semantic-WER以及二元关键性进行比较。在594条源话语中,联合调用准确率从参考转录的82.3%降至三个ASR前端的71.9%-74.9%。数字和否定编辑导致21.9-25.7个百分点的损失,而控制编辑仅造成1.4-2.9个百分点的损失。与二元关键性相比,详细的任务角色标签将按源分组的ROC-AUC从0.73提升至0.78,表明下游影响强烈依赖于受影响的任务角色。

英文摘要:

In cascaded voice agents, ASR errors can have markedly different downstream consequences: a small transcription error may leave a tool call unchanged or alter its intent or arguments. Conventional word error rate (WER), however, does not capture these differences. We introduce SpeechEvalX, a paired diagnostic protocol that evaluates how task roles affected by ASR errors influence structured tool calls. Controlled edits are matched by source utterance, edit count, and operation, while task-role labels are compared with lexical severity, BERT cosine distance, Semantic-WER, and binary criticality. Across 594 source utterances, joint call accuracy falls from 82.3% with reference transcripts to 71.9-74.9% across three ASR front ends. Number and negation edits cause 21.9-25.7 percentage-point losses, versus 1.4-2.9 points for control edits. Detailed task-role labels improve source-grouped ROC-AUC from 0.73 to 0.78 over binary criticality, showing that downstream impact depends strongly on the task role affected.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑