SPEARBench：面向流式语音转语音语言模型的自然度评估基准
SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models
AI总结：
针对现有基准无法评估流式语音转语音模型对话自然度的问题，构建多维度评估基准SPEARBench，可从多维度评测模型，发现当前模型在多类对话行为维度仍与人类存在差距。
摘要：
流式语音转语音语言模型旨在直接用合成语音回应口语查询。但标准语音与文本基准无法衡量这类系统在对话中的自然表现，而对话中的时机、话轮转换、韵律、人际立场、语言与方言一致性、关系适配性会共同影响感知质量。本文提出SPEARBench，这一基准用于从问答交互维度评估语音转语音语言模型的自然度。SPEARBench从Seamless Interaction语料库构建受控对话提示，在多个模型上运行推理，采用覆盖响应延迟、中断情况、语音质量、ASR鲁棒性、语言与方言一致性、情绪自然度、人际立场以及可解释分布基线的多维度协议评估生成答案。该基准包含人类原始回答作为参考对照，已公布多款主流模型的测试结果。结果表明，当前模型可实现高信号级质量与低ASR错误率，但在延迟、重叠、方言留存、情绪适配以及人际立场动态性方面仍与人类对话行为存在差异。
英文摘要：
Streaming speech-to-speech language models aim to answer spoken queries directly with synthetic speech. However, standard speech and text benchmarks do not capture whether these systems behave naturally in conversations, where timing, turn-taking, prosody, interpersonal stance, language and dialect consistency, and relationship-aware appropriateness jointly shape perceived quality. We introduce SPEARBench, a benchmark for evaluating naturalness in speech-to-speech language models from question-answer interactions. SPEARBench constructs controlled dialogue prompts from the Seamless Interaction corpus, runs inference across multiple models, and evaluates generated answers using a multidimensional protocol that covers response latency, interruptions, speech quality, ASR robustness, language and dialect consistency, emotional naturalness, interpersonal stance, and explainable distributional baselines. The benchmark includes original human answers as a reference condition and reports results for several contemporary models. Results show that current models can achieve high signal-level quality and low ASR error while still differing from human conversational behavior in latency, overlap, dialect preservation, emotional adaptation, and interpersonal stance dynamics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
