arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.01016cs.CLcs.AIeess.AS
收藏

PolySpeech-100:面向100多种语言和方言的大规模语音理解基准

PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects

Shenzhen International Graduate School, Tsinghua University(深圳国际研究生院,清华大学);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系);JD AI Research(京东人工智能研究院)

Sicheng Yang, Shulan Ruan, Shiwei Wu, Yu Liu, Lu Fan, Zhi Li, You He

AI总结:

为解决现有语音评估基准在资源丰富语言偏向、缺乏语义推理和忽视方言的问题,提出PolySpeech-100基准,通过混合构建管道覆盖110种语言变体,并评估22个模型,发现开源端到端模型在重方言上优于级联系统,而思维链提示在零样本设置下会降低性能。

摘要:

虽然端到端(E2E)语音大语言模型(Speech-LLMs)正在快速发展,但它们的评估方法仍局限于简单转录的时代。现有基准存在三个关键限制:明显偏向高资源语言、关注低级识别(ASR)而非语义推理,以及忽视区域方言。为弥补这一差距,我们引入了PolySpeech-100,这是一个大规模基准,旨在评估110种语言变体上的“母语级”语音理解。我们采用了一种新颖的混合构建管道,将黄金标准的人类录音与指令驱动的合成语音相结合,从而覆盖了19种不同的中文方言和80多种低资源语言。对22个最先进模型(包括Gemini-3、GPT-Audio和Qwen2.5-Omni)的广泛评估得出了关键见解。首先,我们证明开源端到端模型在重方言上优于级联(ASR+LLM)系统,证明直接音频处理保留了标准转录中经常丢失的关键副语言线索和韵律特征(例如语调、重音)。其次,我们揭示了一个显著的性能差距:虽然商业模型保持稳健,但开源模型在低资源语言上遭受灾难性退化。最后,反直觉的是,我们观察到在标准零样本设置下,思维链提示经常降低大多数评估模型的语音理解性能,揭示了当前架构中潜在的多模态对齐差距。PolySpeech-100为下一代包容性、全能的语音LLM建立了严格标准。数据、演示和代码公开于https://github.com/YoungSeng/PolySpeech-100。

英文摘要:

While End-to-End (E2E) Speech-Large Language Models (Speech-LLMs) are rapidly evolving, their evaluation methodologies remain limited to the era of simple transcription. Existing benchmarks suffer from three critical limitations: a pronounced bias towards high-resource languages, a focus on low-level recognition (ASR) rather than semantic reasoning, and a neglect of regional dialects. To bridge this gap, we introduce PolySpeech-100, a massive-scale benchmark designed to assess `native-level' speech comprehension across 110 linguistic variants. We employ a novel hybrid construction pipeline that augments gold-standard human recordings with instruction-driven synthetic speech, allowing us to cover 19 distinct Chinese dialects and over 80 low-resource languages. Extensive evaluation of 22 state-of-the-art models (including Gemini-3, GPT-Audio, and Qwen2.5-Omni) yields pivotal insights. First, we demonstrate that open-source E2E models outperform Cascade (ASR+LLM) systems on heavy dialects, proving that direct audio processing preserves critical paralinguistic cues and prosodic features (e.g., intonation, stress) that are often lost in standard transcription. Second, we reveal a significant performance gap: while commercial models maintain robustness, open-source models suffer catastrophic degradation on low-resource languages. Finally, counter-intuitively, we observe that under standard zero-shot settings, Chain-of-Thought prompting frequently degrades speech understanding performance for most evaluated models, revealing a potential modality alignment gap in current architectures. PolySpeech-100 establishes a rigorous standard for the next generation of inclusive, omni-capable Speech-LLMs. The data, demo, and code are publicly available at https://github.com/YoungSeng/PolySpeech-100.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑