Recent end-to-end speech language models (SLMs) have expanded upon the capabilities of large language models (LLMs) by incorporating pre-trained speech models. However, these SLMs often undergo extensive speech instruction-tuning to bridge the gap between speech and text modalities. This requires significant annotation efforts and risks catastrophic forgetting of the original language capabilities. In this work, we present a simple yet effective automatic process for creating speech-text pair data that carefully injects speech paralinguistic understanding abilities into SLMs while preserving the inherent language capabilities of the text-based LLM. Our model demonstrates general capabilities for speech-related tasks without the need for speech instruction-tuning data, achieving impressive performance on Dynamic-SUPERB and AIR-Bench-Chat benchmarks. Furthermore, our model exhibits the ability to follow complex instructions derived from LLMs, such as specific output formatting and chain-of-thought reasoning. Our approach not only enhances the versatility and effectiveness of SLMs but also reduces reliance on extensive annotated datasets, paving the way for more efficient and capable speech understanding systems.
S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Models
Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanjing University（南京大学）；Shenzhen Loop Area Institute（深圳河套学院）；CentraleSupélec, Université Paris-Saclay（巴黎萨克雷大学CentraleSupélec分校）；National University of Singapore（新加坡国立大学）
Self-Powered LLM Modality Expansion for Large Speech-Text Models
Institute of Computing and Intelligence, Harbin Institute of Technology（哈尔滨工业大学计算与智能学院）；Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；The University of Sydney（悉尼大学）；Nanyang Technological University（南洋理工大学）
Enhancing Speech Instruction Understanding and Disambiguation in Robotics via Speech Prosody
IT University of Copenhagen（丹麦技术大学）；University of Florida（佛罗里达大学）；Brown University（布朗大学）；Intelligent Agents Research Group（智能代理研究组）；Intelligent Robot Lab（智能机器人实验室）
Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context
School of Data Science, Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院、深圳大数据研究院）；Bytedance（字节跳动）
Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision
College of Computing and Data Science（计算与数据科学学院）；Nanyang Technological University（南洋理工大学）；AI Singapore（AI新加坡）；National University of Singapore（新加坡国立大学）；Institute for Infocomm Research (I$^2$R)（信息通信研究所）；A$^{*}$STAR（A*星）
Audio-FLAN: An Instruction-Following Dataset for Unified Audio Understanding and Generation of Speech, Music, and Sound
The Hong Kong University of Science（香港科学与技术大学）；Inner Mongolia University（内蒙古大学）；Beihang University（北京航空航天大学）；Queen Mary University of London（伦敦玛丽女王大学）；The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）；University of Surrey（萨里大学）；University of Rochester（罗切斯特大学）；Independent Researcher（独立研究者）
From Specialization to Generalization: Instruction-tuned LLMs for Robust Harmful Content Mitigation
School of Computation, Information and Technology, TU Munich（慕尼黑工业大学计算、信息与技术学院）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Munich Data Science Institute（慕尼黑数据科学研究所）
LLaMA-Omni: Seamless Speech Interaction with Large Language Models
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Key Laboratory of AI Safety, Chinese Academy of Sciences（中国科学院人工智能安全重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Do What I Say: A Spoken Prompt Dataset for Instruction-Following
Karlsruhe Institute of Technology, Germany（卡尔斯鲁厄理工学院，德国）；Fondazione Bruno Kessler, Italy（布鲁诺·凯瑟尔基金会，意大利）；ACC Cyfronet AGH, Poland（AGH计算机科学与技术学院，波兰）；AGH University of Krakow, Poland（克拉科夫AGH大学，波兰）；Carnegie Mellon University, U.S.（卡内基梅隆大学，美国）
Rapidly Developing High-quality Instruction Data and Evaluation Benchmark for Large Language Models with Minimal Human Effort: A Case Study on Japanese
FreezeEmpath: Efficient Training for Empathetic Spoken Chatbots with Frozen LLMs
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences（智能信息处理重点实验室，计算技术研究所，中国科学院）；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）
