TextME: Bridging Unseen Modalities Through Text Descriptions
Department of Artificial Intelligence, Ajou University, Suwon, South Korea；Division of Language \& AI, Hankuk University of Foreign Studies, Seoul, Korea；Graduate School of AI, POSTECH, Pohang, Korea；Department of Software, Ajou University, Suwon, South Korea
Soyeon Hong, Jinchan Kim, Jaegook You, Seungtaek Choi, Suha Kwak, Hyunsouk Cho
Expanding multimodal representations to novel modalities is constrained by reliance on large-scale paired datasets (e.g., text-image, text-audio, text-3D, text-molecule), which are costly and often infeasible in domains requiring expert annotation such as medical imaging and molecular analysis. We introduce TextME, the first text-only modality expansion framework, to the best of our knowledge, projecting diverse modalities into LLM embedding space as a unified anchor. Our approach exploits the geometric structure of pretrained contrastive encoders to enable zero-shot cross-modal transfer using only text descriptions, without paired supervision. We empirically validate that such consistent modality gaps exist across image, video, audio, 3D, X-ray, and molecular domains, demonstrating that text-only training can preserve substantial performance of pretrained encoders. We further show that our framework enables emergent cross-modal retrieval between modality pairs not explicitly aligned during training (e.g., audio-to-image, 3D-to-image). These results establish text-only training as a practical alternative to paired supervision for modality expansion.
ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Bytedance Inc.（字节跳动公司）
Bridging Modalities: Knowledge Distillation and Masked Training for Translating Multi-Modal Emotion Recognition to Uni-Modal, Speech-Only Emotion Recognition
Locate Then Generate: Bridging Vision and Language with Bounding Box for Scene-Text VQA
School of Data Science, University of Science and Technology of China（中国科学技术大学数据科学学院）；School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；Tencent YouTu Lab（腾讯优图实验室）
Bridging Collaborative Filtering and Large Language Models with Dynamic Alignment, Multimodal Fusion and Evidence-grounded Explanations
Department of Software \& Microelectronics, Peking University, Beijing, China；Economic Law School, China University of Political Science；Financial Media, Peking University, ChangSha, China
End-to-end optimization has achieved state-of-the-art performance on many specific problems, but there is no straight-forward way to combine pretrained models for new problems. Here, we explore improving modularity by learning a post-hoc interface between two existing models to solve a new task. Specifically, we take inspiration from neural machine translation, and cast the challenging problem of cross-modal domain transfer as unsupervised translation between the latent spaces of pretrained deep generative models. By abstracting away the data representation, we demonstrate that it is possible …
How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
College of Computer, National University of Defense Technology（国防科技大学计算机学院）；School of Computer and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）
CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning
School of Electronics and Information Technology, Sun Yat-Sen University（中山大学电子与信息学院）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；School of Computer Science, South China Normal University（华南师范大学计算机科学学院）；Desay SV Automotive Co., Ltd（德赛股份有限公司）；Pazhou Laboratory（琶洲实验室）
EmotionRankCLAP: Bridging Natural Language Speaking Styles and Ordinal Speech Emotion via Rank-N-Contrast
Center for Language and Speech Processing (CLSP), Johns Hopkins UniversityUSA（语言与语音处理中心（CLSP），约翰霍普金斯大学）；The University of Texas at Dallas, USA（德克萨斯大学达拉斯分校）；Amazon, USA（亚马逊）；NUSSingapore（南洋理工大学）；Language Technologies Institute (LTI), Carnegie Mellon UniversityUSA（语言技术研究所（LTI），卡内基梅隆大学）
Harmony: A Unified Framework for Modality Incremental Learning
Peng Cheng Laboratory（鹏城实验室）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）
