arXiv 2511.092322026-03-31cs.CLcs.SD

POTSA：一种用于语音到文本翻译的跨语言语音对齐框架

POTSA: A Cross-Lingual Speech Alignment Framework for Speech-to-Text Translation

Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University（天津大学认知计算与应用天津市重点实验室）；Nanyang Technological University（南洋理工大学）；Huiyan Technology Company, Ltd.（慧眼科技有限公司）；Chinese Academy of Sciences（中国科学院）；Tibet University（西藏大学）

Xuanchen Li, Chenrui Cui, Tianrui Wang, Meng Ge, Zikang Huang, Yizhou Peng, Jin Li, Yuheng Lu, Yu Jiang, Nyima Tashi, Longbiao Wang, Jianwu Dang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出POTSA框架，通过最优传输和跨语言平行语音对缓解多语言语音翻译中的语义偏见问题，实验显示在FLEURS数据集上取得SOTA性能，提升BLEU分数。

摘要：

语音大语言模型在多语言语音到文本翻译中取得了突破。然而，现有方法常忽视源语言间的语义共性，导致翻译性能偏倚。本文提出POTSA（Parallel Optimal Transport for Speech Alignment），基于跨语言平行语音对和最优传输，旨在弥合高、低资源翻译差距。首先，引入偏倚补偿模块粗对齐初始语音表示。其次，利用平行对在Q-Former上施加token级OT约束，建立细粒度表示一致性。然后，应用层调度策略，将OT约束聚焦于语义有益的层。实验表明，在FLEURS数据集上，本方法取得SOTA性能，相较于五种常见语言提升1.29 BLEU，零样本语言提升2.93 BLEU，仅使用每种语言10小时的平行语音。

英文摘要：

Speech Large Language Models have achieved breakthroughs in multilingual speech-to-text translation. However, existing approaches often overlook semantic commonalities across source languages, leading to biased translation performance. In this work, we propose POTSA (Parallel Optimal Transport for Speech Alignment), a new framework based on cross-lingual parallel speech pairs and Optimal Transport, designed to bridge high- and low-resource translation gaps. First, we introduce a Bias Compensation module to coarsely align initial speech representations. Second, we impose token-level OT constraints on a Q-Former using parallel pairs to establish fine-grained representation consistency. Then, we apply a layer scheduling strategy to focus OT constraints on semantically beneficial layers. Experiments on FLEURS show our method achieves SOTA performance, with +1.29 BLEU over five common languages and +2.93 BLEU on zero-shot languages, using only 10 hours of parallel speech per language.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。