arXiv 2509.177652025-09-22cs.CLcs.AIcs.CVeess.AS

Qwen3-Omni 技术报告

Qwen3-Omni Technical Report

Qwen Team（通义实验室）

Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Lin… 展开作者

Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo Zheng, Rui Men, Fan Zhou, Bowen Yu, Jianxin Yang, Le Yu, Jingren Zhou, Junyang Lin

备注：: https://github.com/QwenLM/Qwen3-Omni

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出单一多模态模型 Qwen3-Omni，基于 Thinker-Talker MoE 架构统一文本、图像、音频和视频感知与生成，在多项音频基准上取得 SOTA，并通过多码本与轻量因果 ConvNet 实现低延迟流式语音输出。

摘要：

我们提出 Qwen3-Omni，这是一个单一多模态模型，首次在文本、图像、音频和视频上保持最先进的性能，且相对于单模态对应模型没有任何性能下降。Qwen3-Omni 与 Qwen 系列中同尺寸的单模态模型性能相当，尤其在音频任务上表现突出。在 36 个音频和音视频基准测试中，Qwen3-Omni 在 32 个基准上取得开源 SOTA，在 22 个基准上取得整体 SOTA，超越了 Gemini-2.5-Pro、Seed-ASR 和 GPT-4o-Transcribe 等强大的闭源模型。Qwen3-Omni 采用 Thinker-Talker MoE 架构，统一了文本、图像、音频和视频的感知与生成，能够产生流畅的文本和自然的实时语音。它支持 119 种语言的文本交互、19 种语言的语音理解和 10 种语言的语音生成。为了降低流式合成中的首包延迟，Talker 使用多码本方案自回归地预测离散语音编解码器。利用这些码本的表示能力，我们用轻量级因果 ConvNet 替代了计算密集型的逐块扩散，从而能够从第一个编解码帧开始进行流式输出。在冷启动设置下，Qwen3-Omni 实现了 234 毫秒的理论端到端首包延迟。为了进一步加强多模态推理，我们引入了一个 Thinking 模型，该模型可对任意模态的输入进行显式推理。由于研究界目前缺乏通用的音频描述模型，我们对 Qwen3-Omni-30B-A3B 进行了微调，得到 Qwen3-Omni-30B-A3B-Captioner，该模型能够为任意音频输入生成详细且低幻觉的描述。Qwen3-Omni-30B-A3B、Qwen3-Omni-30B-A3B-Thinking 和 Qwen3-Omni-30B-A3B-Captioner 已在 Apache 2.0 许可证下公开发布。

英文摘要：

We present Qwen3-Omni, a single multimodal model that, for the first time, maintains state-of-the-art performance across text, image, audio, and video without any degradation relative to single-modal counterparts. Qwen3-Omni matches the performance of same-sized single-modal models within the Qwen series and excels particularly on audio tasks. Across 36 audio and audio-visual benchmarks, Qwen3-Omni achieves open-source SOTA on 32 benchmarks and overall SOTA on 22, outperforming strong closed-source models such as Gemini-2.5-Pro, Seed-ASR, and GPT-4o-Transcribe. Qwen3-Omni adopts a Thinker-Talker MoE architecture that unifies perception and generation across text, images, audio, and video, yielding fluent text and natural real-time speech. It supports text interaction in 119 languages, speech understanding in 19 languages, and speech generation in 10 languages. To reduce first-packet latency in streaming synthesis, Talker autoregressively predicts discrete speech codecs using a multi-codebook scheme. Leveraging the representational capacity of these codebooks, we replace computationally intensive block-wise diffusion with a lightweight causal ConvNet, enabling streaming from the first codec frame. In cold-start settings, Qwen3-Omni achieves a theoretical end-to-end first-packet latency of 234 ms. To further strengthen multimodal reasoning, we introduce a Thinking model that explicitly reasons over inputs from any modality. Since the research community currently lacks a general-purpose audio captioning model, we fine-tuned Qwen3-Omni-30B-A3B to obtain Qwen3-Omni-30B-A3B-Captioner, which produces detailed, low-hallucination captions for arbitrary audio inputs. Qwen3-Omni-30B-A3B, Qwen3-Omni-30B-A3B-Thinking, and Qwen3-Omni-30B-A3B-Captioner are publicly released under the Apache 2.0 license.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。