Recent advancements in speech large language models (SpeechLLMs) have attracted considerable attention. Nonetheless, current methods exhibit suboptimal performance in adhering to speech instructions. Notably, the intelligence of models significantly diminishes when processing speech-form input as compared to direct text-form input. Prior work has attempted to mitigate this semantic inconsistency between speech and text representations through techniques such as representation and behavior alignment, which involve the meticulous design of data pairs during the post-training phase. In this paper, we introduce a simple and scalable training method called InSerter, which stands for Interleaved Speech-Text Representation Pre-training. InSerter is designed to pre-train large-scale unsupervised speech-text sequences, where the speech is synthesized from randomly selected segments of an extensive text corpus using text-to-speech conversion. Consequently, the model acquires the ability to generate textual continuations corresponding to the provided speech segments, obviating the need for intensive data design endeavors. To systematically evaluate speech instruction-following capabilities, we introduce SpeechInstructBench, the first comprehensive benchmark specifically designed for speech-oriented instruction-following tasks. Our proposed InSerter achieves SOTA performance in SpeechInstructBench and demonstrates superior or competitive results across diverse speech processing tasks.
An Evaluation of Interleaved Instruction Tuning on Semantic Reasoning Performance in an Audio MLLM
The Graduate Center, CUNY（CUNY研究生中心）；Brooklyn College, CUNY（CUNY布鲁克林学院）；Borough of Manhattan Community College, CUNY（CUNY曼哈顿社区学院）；The City College of New York, CUNY（CUNY纽约城市学院）
Modality-Specialized Synergizers for Interleaved Vision-Language Generalists
Virginia Tech（弗吉尼亚理工大学）；Intuit AI Research（Intuit AI研究院）；Meta AI；The Chinese University of Hong Kong（香港中文大学）；University of California, Davis（加州大学戴维斯分校）
CodeEvo: Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback
Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；New York University（纽约大学）；Carnegie Mellon University（卡内基梅隆大学）；Shanghai Innovation Institute（上海创新研究院）
CX-Mind: A Pioneering Multimodal Large Language Model for Interleaved Reasoning in Chest X-ray via Curriculum-Guided Reinforcement Learning
College of Health Science and Technology, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院健康科学与技术学院）；Shanghai Innovation Institute（上海创新研究院）；Clinical Center for Sports Medicine, Department of Orthopaedics, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院骨科临床中心）；School of Basic Medical Sciences, Intelligent Medicine Institute, Fudan University（复旦大学基础医学学院）；Department of Hematology, The First Affiliated Hospital, College of Medicine, Zhejiang University（浙江大学医学院第一附属医院血液科）；MoE Key Laboratory of Brain-Inspired Intelligent Perception and Cognition, University of Science and Technology of China（中国科学技术大学脑启发智能感知与认知教育部重点实验室）；Department of Public Health and Primary Care, University of Cambridge（剑桥大学公共卫生与初级保健学院）；Department of Medicine, Faculty of Health Sciences, Universidad CEU Cardenal Herrera（CEU卡德纳尔-赫尔曼大学健康科学学院医学系）；Faculty of Medicine, University of Helsinki（赫尔辛基大学医学院）；X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院X-LANCE实验室）；Department of Hepatobiliary Surgery, National Cancer Center / National Clinical Research Center for Cancer / Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院肿瘤医院肝胆外科）；Department of Surgery, The Ohio State University Wexner Medical Center, The James Comprehensive Cancer Center（俄亥俄州立大学韦克斯纳医学中心外科部，詹姆斯综合癌症中心）；Ningbo Institute of Technology, Beihang University（北航宁波理工学院）
Beihang University（北航）；M-A-P；The Hong Kong Polytechnic University（香港理工大学）；AIWaves；University of Alberta（阿尔伯塔大学）；University of Waterloo（滑铁卢大学）；University of Manchester（曼彻斯特大学）；Chinese Academy of Sciences（中国科学院）；Peking University（北京大学）；Shanghai AI Lab（上海AI实验室）；Nanjing University（南京大学）；Kuaishou Technology（快手科技）
Interleaving Reasoning for Better Text-to-Image Generation
East China Normal University（华东师范大学）；The Chinese University of Hong Kong（香港中文大学）；Xiaohongshu Inc.（小红书公司）；University of California, Los Angeles（加州大学洛杉矶分校）；Zhejiang University（浙江大学）；University of Oxford（牛津大学）
Efficient Training for Cross-lingual Speech Language Models
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)（中国科学院计算技术研究所智能信息处理重点实验室）；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所人工智能安全国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）
