arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.04711cs.SDcs.AI
收藏

基于大语音模型的语义通信

Large Speech Model Enabled Semantic Communication

School of Electronics, Peking University(北京大学电子学院);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系);Department of Electrical and Electronic Engineering, The University of Hong Kong(香港大学电子与电气工程系);Department of Computer Science and Engineering, Kyung Hee University(庆熙大学计算机科学与工程系)

Yun Tian, Zhijin Qin, Guocheng Lv, Ye Jin, Kaibin Huang, Zhu Han

AI总结:

本文提出基于大语音模型的语义通信系统,利用Mimi编解码器和LoRA微调技术,实现适应性压缩与鲁棒传输,支持低带宽下的高质量语音传输。

摘要:

现有的语音语义通信系统主要基于联合源-信道编码(JSCC)架构,已展现出出色的性能,但其效果仍受到专门针对特定任务和数据集设计的模型结构的限制。最近的进展表明,预训练在大规模数据集上的生成式大模型可以在少量微调的情况下,在多种下游任务上表现出色。为了利用大模型中丰富的语义知识并实现适应性传输,我们提出了一种基于大语音模型的语义通信(LargeSC)系统。同时实现适应性压缩和在失真信道上的鲁棒传输仍具有挑战性,需要在压缩效率、语音质量和延迟之间进行权衡。在本工作中,我们采用Mimi作为语音编解码器,将语音转换为与现有网络架构兼容的离散标记。我们提出一个自适应控制器模块,能够实现自适应传输和带内不等错误保护(UEP),动态适应语音内容和包丢失概率在带宽限制下的情况。此外,我们采用低秩适应(LoRA)对Moshi基础模型进行微调,以实现丢失语音标记的生成恢复。仿真结果表明,所提出的系统支持550 bps到2.06 kbps的带宽范围,在高包丢失率下优于传统基线,在端到端延迟方面达到约460 ms,从而证明了其在实时部署中的潜力。

英文摘要:

Existing speech semantic communication systems mainly based on Joint Source-Channel Coding (JSCC) architectures have demonstrated impressive performance, but their effectiveness remains limited by model structures specifically designed for particular tasks and datasets. Recent advances indicate that generative large models pre-trained on massive datasets, can achieve outstanding performance arexhibit exceptional performance across diverse downstream tasks with minimal fine-tuning. To exploit the rich semantic knowledge embedded in large models and enable adaptive transmission over lossy channels, we propose a Large Speech Model enabled Semantic Communication (LargeSC) system. Simultaneously achieving adaptive compression and robust transmission over lossy channels remains challenging, requiring trade-offs among compression efficiency, speech quality, and latency. In this work, we employ the Mimi as a speech codec, converting speech into discrete tokens compatible with existing network architectures. We propose an adaptive controller module that enables adaptive transmission and in-band Unequal Error Protection (UEP), dynamically adjusting to both speech content and packet loss probability under bandwidth constraints. Additionally, we employ Low-Rank Adaptation (LoRA) to finetune the Moshi foundation model for generative recovery of lost speech tokens. Simulation results show that the proposed system supports bandwidths ranging from 550 bps to 2.06 kbps, outperforms conventional baselines in speech quality under high packet loss rates and achieves an end-to-end latency of approximately 460 ms, thereby demonstrating its potential for real-time deployment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑