Recent advancements in large language models (LLMs) and multimodal speech-text models have laid the groundwork for seamless voice interactions, enabling real-time, natural, and human-like conversations. Previous models for voice interactions are categorized as native and aligned. Native models integrate speech and text processing in one framework but struggle with issues like differing sequence lengths and insufficient pre-training. Aligned models maintain text LLM capabilities but are often limited by small datasets and a narrow focus on speech tasks. In this work, we introduce MinMo, a Multimodal Large Language Model with approximately 8B parameters for seamless voice interaction. We address the main limitations of prior aligned multimodal models. We train MinMo through multiple stages of speech-to-text alignment, text-to-speech alignment, speech-to-speech alignment, and duplex interaction alignment, on 1.4 million hours of diverse speech data and a broad range of speech tasks. After the multi-stage training, MinMo achieves state-of-the-art performance across various benchmarks for voice comprehension and generation while maintaining the capabilities of text LLMs, and also facilitates full-duplex conversation, that is, simultaneous two-way communication between the user and the system. Moreover, we propose a novel and simple voice decoder that outperforms prior models in voice generation. The enhanced instruction-following capabilities of MinMo supports controlling speech generation based on user instructions, with various nuances including emotions, dialects, and speaking rates, and mimicking specific voices. For MinMo, the speech-to-text latency is approximately 100ms, full-duplex latency is approximately 600ms in theory and 800ms in practice. The MinMo project web page is https://funaudiollm.github.io/minmo, and the code and models will be released soon.
MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；National Taiwan University（国立台湾大学）；Massachusetts Institute of Technology（麻省理工学院）；Carnegie Mellon University（卡内基梅隆大学）；Meta AI
OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction
Tongji University（同济大学）；Tongyi Laboratory（通义实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）
MOMO: A framework for seamless physical, verbal, and graphical robot skill learning and adaptation
German Aerospace Center (DLR), Institute of Robotics and Mechatronics (RMC)（德国航空航天中心（DLR）机器人与机电研究所）；School of Computation, Information and Technology (CIT), Technical University of Munich (TUM)（计算、信息与技术学院（CIT），慕尼黑技术大学）
Sparks of Large Audio Models: A Survey and Outlook
Queensland University of Technology（昆士兰科技大学）；Emulation AI；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；NUCES（国立计算机与新兴科学大学）；Bytedance AI Lab（字节跳动人工智能实验室）；University of Lincoln（林肯大学）；University of Surrey（萨里大学）；Ping An Technology（平安科技）；University of Western Australia（西澳大学）；Nanyang Technological University（南洋理工大学）；Imperial College London（伦敦帝国学院）；University of Augsburg（奥格斯堡大学）
Audio Jailbreak Attacks: Exposing Vulnerabilities in SpeechGPT in a White-Box Framework
Department of Computer Science University of Missouri-Kansas City（计算机科学系 密苏里大学-康科特分校）；Department of Computer Science and Physics Rider University（计算机科学与物理系 Rider大学）；Department of Electrical and Computer Engineering University of Hawai’i at Mānoa（电气与计算机工程系 夏威夷大学马诺亚分校）
ESPnet-SpeechLM: An Open Speech Language Model Toolkit
Carnegie Mellon University（卡内基梅隆大学）；Mitsubishi Electric Research Laboratories（三菱电机研究所）；LY Corporation（LY公司）；Renmin University of China（中国人民大学）；Brno University of Technology（布尔诺理工大学）；NVIDIA Research（英伟达研究院）
