Mind the Gap No More: Achieving Zero-Gap Multimodal Integration via One Tokenizer
Research Center for Frontier Fundamental Studies, Zhejiang Lab（前沿基础研究研究中心，浙江实验室）；Research Center for Scientific Data Hub, Zhejiang Lab（科学数据枢纽研究中心，浙江实验室）
Yanan Li, Christina Yi Jin, Yuan Jin, Manli Luo, Tie Xu, Shuai Jiao, Wei He, Qing Zhang
A central challenge in developing Multimodal Large Language Models (MLLMs) is effectively integrating heterogeneous inputs into a cohesive reasoning engine. Current paradigms predominantly rely on modular architectures that introduce modality-specific encoders and cross-modal fusion mechanisms. However, these designs are fundamentally bottlenecked by a geometric modality gap, forcing the LLM to expend significant computational capacity on geometric reconciliation rather than deep cross-modal reasoning. In this work, we formally characterize this modality gap and theoretically demonstrate that native architectures, specifically those employing a unified vocabulary, intrinsically maintain a zero-gap state across all hidden layers. Guided by these theoretical findings, we propose \textit{One Tokenizer}, a native architecture that maps all modalities directly into a shared token space. We empirically validate this framework on a DNA--text multimodal testbed. Our extensive evaluations reveal that by achieving seamless integration within the LLM's native latent space, One Tokenizer consistently outperforms encoder-based modular counterparts, providing a fundamentally superior framework for deep biological reasoning.
CoMMET: A Psychologically Grounded Benchmark for Evaluating Theory of Mind in Multimodal LLMs
Agency for Science, Technology and Research (A*STAR)（科技研究局）；Nanyang Technological University（南洋理工大学）；Hong Kong University of Science and Technology (Guangzhou), China（香港科技大学（广州），中国）
Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing
State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Department of Computer Science, The University of Manchester（曼彻斯特大学计算机科学系）；Department of Language Science and Technology, Hong Kong Polytechnic University（香港理工大学语言科学与技术系）
$μ^2$Tokenizer: Differentiable Multi-Scale Multi-Modal Tokenizer for Radiology Report Generation
School of Electronic Engineering and Computer Science（电子工程与计算机科学学院）；Queen Mary University of London（伦敦女王学院）；School of Engineering, College of Engineering and Physical Sciences（工程学院，工程与物理科学学院）；University of Birmingham（伯明翰大学）；Guangdong University of Technology（广东工业大学）；Meta Inc. US（Meta美国公司）；Nuffield Department of Clinical Neurosciences（临床神经科学系）；University of Oxford（牛津大学）；William Harvey Research Institute, NIHR Barts Biomedical Research Centre, Queen Mary University London（威廉·哈里弗研究所在NIHR巴茨生物医学研究中心，伦敦女王学院）
From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
School of Computer Science, Peking University（北京大学计算机学院）；The Chinese University of Hong Kong（香港中文大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
CX-Mind: A Pioneering Multimodal Large Language Model for Interleaved Reasoning in Chest X-ray via Curriculum-Guided Reinforcement Learning
College of Health Science and Technology, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院健康科学与技术学院）；Shanghai Innovation Institute（上海创新研究院）；Clinical Center for Sports Medicine, Department of Orthopaedics, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院骨科临床中心）；School of Basic Medical Sciences, Intelligent Medicine Institute, Fudan University（复旦大学基础医学学院）；Department of Hematology, The First Affiliated Hospital, College of Medicine, Zhejiang University（浙江大学医学院第一附属医院血液科）；MoE Key Laboratory of Brain-Inspired Intelligent Perception and Cognition, University of Science and Technology of China（中国科学技术大学脑启发智能感知与认知教育部重点实验室）；Department of Public Health and Primary Care, University of Cambridge（剑桥大学公共卫生与初级保健学院）；Department of Medicine, Faculty of Health Sciences, Universidad CEU Cardenal Herrera（CEU卡德纳尔-赫尔曼大学健康科学学院医学系）；Faculty of Medicine, University of Helsinki（赫尔辛基大学医学院）；X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院X-LANCE实验室）；Department of Hepatobiliary Surgery, National Cancer Center / National Clinical Research Center for Cancer / Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院肿瘤医院肝胆外科）；Department of Surgery, The Ohio State University Wexner Medical Center, The James Comprehensive Cancer Center（俄亥俄州立大学韦克斯纳医学中心外科部，詹姆斯综合癌症中心）；Ningbo Institute of Technology, Beihang University（北航宁波理工学院）
