arXiv 2510.045932026-09-07eess.AScs.SD

UniVoice: 统一自回归ASR与基于流匹配的TTS的大语言模型框架

UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models

Xiamen University, China（厦门大学）；Shanghai Innovation Institute, China（上海创新研究院）；Shanghai Jiao Tong University, China（上海交通大学）；Zhejiang University, China（浙江大学）

Wenhao Guan, Zhikang Niu, Ziyue Jiang, Kaidi Wang, Peijie Chen, Qingyang Hong, Lin Li, Xie Chen

备注：: accepted at interspeech2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出UniVoice，通过连续表示统一语音识别与合成，结合自回归建模和流匹配，设计双重注意力机制解决模态差异，实现高质量零样本语音克隆。

摘要：

大语言模型在自动语音识别和文本转语音系统中展现出有前景的性能，逐渐成为主流方法。然而，当前大多数方法分别处理这两个任务，而非通过统一框架。本工作旨在将这两个任务集成到一个统一模型中。尽管离散语音标记化能够实现联合建模，但其固有的信息损失限制了识别和生成的性能。在本工作中，我们提出了UniVoice，一个通过连续表示的统一大语言模型框架，无缝地将语音识别和合成集成在单个模型中。我们的方法结合了自回归建模在语音识别中的优势与流匹配在高品质生成中的优势。为了缓解自回归模型和流匹配模型之间的固有差异，我们进一步设计了一种双重注意力机制，在因果掩码（用于识别）和双向注意力掩码（用于合成）之间切换。此外，所提出的文本前缀条件语音填充方法实现了高保真度的零样本语音克隆。实验结果表明，我们的方法在ASR和零样本TTS任务中能够达到或超越当前单任务建模方法。本工作探索了端到端语音理解和生成的新可能性。代码可在该 https URL 获取。

英文摘要：

Large language models (LLMs) have demonstrated promising performance in both automatic speech recognition (ASR) and text-to-speech (TTS) systems, gradually becoming the mainstream approach. However, most current approaches address these tasks separately rather than through a unified framework. This work aims to integrate these two tasks into one unified model. Although discrete speech tokenization enables joint modeling, its inherent information loss limits performance in both recognition and generation. In this work, we present UniVoice, a unified LLM framework through continuous representations that seamlessly integrates speech recognition and synthesis within a single model. Our approach combines the strengths of autoregressive modeling for speech recognition with flow matching for high-quality generation. To mitigate the inherent divergence between autoregressive and flow-matching models, we further design a dual attention mechanism, which switches between a causal mask for recognition and a bidirectional attention mask for synthesis. Furthermore, the proposed text-prefix-conditioned speech infilling method enables high-fidelity zero-shot voice cloning. Experimental results demonstrate that our method can achieve or exceed current single-task modeling methods in both ASR and zero-shot TTS tasks. This work explores new possibilities for end-to-end speech understanding and generation. Code is available at https://github.com/gwh22/UniVoice.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。