School of Electronics, Peking University(北京大学电子学院);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系);Department of Electrical and Electronic Engineering, The University of Hong Kong(香港大学电子与电气工程系);Department of Computer Science and Engineering, Kyung Hee University(庆熙大学计算机科学与工程系)
Yun Tian, Zhijin Qin, Guocheng Lv, Ye Jin, Kaibin Huang, Zhu Han
Existing speech semantic communication systems mainly based on Joint Source-Channel Coding (JSCC) architectures have demonstrated impressive performance, but their effectiveness remains limited by model structures specifically designed for particular tasks and datasets. Recent advances indicate that generative large models pre-trained on massive datasets, can achieve outstanding performance arexhibit exceptional performance across diverse downstream tasks with minimal fine-tuning. To exploit the rich semantic knowledge embedded in large models and enable adaptive transmission over lossy channels, we propose a Large Speech Model enabled Semantic Communication (LargeSC) system. Simultaneously achieving adaptive compression and robust transmission over lossy channels remains challenging, requiring trade-offs among compression efficiency, speech quality, and latency. In this work, we employ the Mimi as a speech codec, converting speech into discrete tokens compatible with existing network architectures. We propose an adaptive controller module that enables adaptive transmission and in-band Unequal Error Protection (UEP), dynamically adjusting to both speech content and packet loss probability under bandwidth constraints. Additionally, we employ Low-Rank Adaptation (LoRA) to finetune the Moshi foundation model for generative recovery of lost speech tokens. Simulation results show that the proposed system supports bandwidths ranging from 550 bps to 2.06 kbps, outperforms conventional baselines in speech quality under high packet loss rates and achieves an end-to-end latency of approximately 460 ms, thereby demonstrating its potential for real-time deployment.
Is Semantics Enough for Speech Mean Opinion Score Prediction?
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China(中国科学技术大学国家语音与语言信息处理工程研究中心)
Speaking Clearly: A Simplified Whisper-Based Codec for Low-Bitrate Speech Coding
School of Computer Science and Artificial Intelligence(计算机科学与人工智能学院);Wuhan University of Technology(武汉理工大学);NEC Corporation(日本电气株式会社);The Hong Kong Polytechnic University(香港理工大学)
Modeling Sarcastic Speech: Semantic and Prosodic Cues in a Speech Synthesis Framework
Speech Technology Lab, University of Groningen, Campus Fryslân, The Netherlands(格罗宁根大学弗里赛兰校区语音技术实验室,荷兰);Center for Language and Cognition, University of Groningen, The Netherlands(格罗宁根大学语言与认知中心,荷兰)
SARA: A Dual-Stream VAE for High-Fidelity Speech Generation via Integrating Semantic and Acoustic Representations
School of Informatics, Xiamen University(厦门大学信息学院);School of Electronic Science and Engineering, Xiamen University(厦门大学电子科学与技术学院);DiDi Global Inc.(滴滴全球股份有限公司)
FuseCodec: Semantic-Contextual Fusion and Supervision for Neural Codecs
Center for Computational & Data Sciences, Independent University, Bangladesh(计算与数据科学中心,独立大学,孟加拉国);Amazon GenAI(亚马逊生成人工智能);Qatar Computing Research Institute(卡塔尔计算研究所);University of Virginia(弗吉尼亚大学)
HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesis
Generative Semantic Communication for Text-to-Speech Synthesis
Shenzhen Future Network of Intelligence Institute(深圳未来网络智能研究院);Guangdong Provincial Key Laboratory of Future Networks of Intelligence, The Chinese University of Hong Kong (Shenzhen)(广东省未来网络智能重点实验室,香港中文大学(深圳));School of Automation, Central South University(中南大学自动化学院)
Parallel GPT: Harmonizing the Independence and Interdependence of Acoustic and Semantic Information for Zero-Shot Text-to-Speech
School of Future Technology, South China University of Technology(未来技术学院,华南理工大学);School of Electronic and Information, South China University of Technology(电子与信息学院,华南理工大学)
Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representations
School of Information and Communication Engineering, Communication University of China(信息与通信工程学院,中国传媒大学);Microsoft Research Asia(微软亚洲研究院);State Key Laboratory of Media Convergence and Communication, Communication University of China(媒体融合与通信国家重点实验室,中国传媒大学)
Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment
School of Informatics, Xiamen University, China(厦门大学信息学院);Huawei Translation Services Center, Beijing, China(华为翻译服务中心);NLP 2 CT Lab, Department of Computer and Information Science, University of Macau(澳门大学计算机与信息科学系NLP 2 CT实验室)
Modeling speech recognition and synthesis simultaneously: Encoding and decoding lexical and sublexical semantic information into speech with no direct access to speech data