本研究论文解决了多模态学习中的模态不匹配挑战，即推理时可用的模态与训练时可用的模态存在差异。我们提出了面向多模态学习的以文本为中心的对齐（Text-centric Alignment for Multi-Modality Learning，TAMML）方法，这是一种创新方法，利用具备上下文学习能力的大语言模型（Large Language Models，LLMs）和基础模型来提升多模态系统在上述条件下的泛化能力。通过利用文本作为统一语义空间的独特属性，TAMML在处理未见过的、多样的且不可预测的模态组合方面展现出显著提升。TAMML不仅能够适配不同的模态，还能保持稳健的性能，彰显了基础模型在克服传统固定模态框架在嵌入表示方面局限性的潜力。本研究为模态可用性动态且不确定的真实世界应用提供了一种灵活有效的解决方案，为该领域做出了贡献。
英文摘要：
This research paper addresses the challenge of modality mismatch in multimodal learning, where the modalities available during inference differ from those available at training. We propose the Text-centric Alignment for Multi-Modality Learning (TAMML) approach, an innovative method that utilizes Large Language Models (LLMs) with in-context learning and foundation models to enhance the generalizability of multimodal systems under these conditions. By leveraging the unique properties of text as a unified semantic space, TAMML demonstrates significant improvements in handling unseen, diverse, and unpredictable modality combinations. TAMML not only adapts to varying modalities but also maintains robust performance, showcasing the potential of foundation models in overcoming the limitations of traditional fixed-modality frameworks in embedding representations. This study contributes to the field by offering a flexible, effective solution for real-world applications where modality availability is dynamic and uncertain.
Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhongke Fanyu Technology Co., Ltd（中科泛语科技有限公司）
Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback
Institute for AI, Peking University（北京大学人工智能研究院）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院）；Huawei Noah’s Ark LAB（华为诺亚方舟实验室）；Baichuan Inc.（百川智能）；Taobao & Tmall Group of Alibaba（阿里巴巴淘宝天猫集团）
Cell segmentation is a fundamental task for computational biology analysis. Identifying the cell instances is often the first step in various downstream biomedical studies. However, many cell segmentation algorithms, including the recently emerging deep learning-based methods, still show limited generality under the multi-modality environment. Weakly Supervised Cell Segmentation in Multi-modality High-Resolution Microscopy Images was hosted at NeurIPS 2022 to tackle this problem. We propose MEDIAR, a holistic pipeline for cell instance segmentation under multi-modality in this challenge. MEDIA…
Singapore University of Technology and Design（新加坡科技设计大学）；National University of Singapore（新加坡国立大学）
Existing multimodal tasks mostly target at the complete input modality setting, i.e., each modality is either complete or completely missing in both training and test sets. However, the randomly missing situations have still been underexplored. In this paper, we present a novel approach named MM-Align to address the missing-modality inference problem. Concretely, we propose 1) an alignment dynamics learning module based on the theory of optimal transport (OT) for indirect missing data imputation; 2) a denoising training algorithm to simultaneously enhance the imputation results and backbone ne…
Multimodal-Guided Dynamic Dataset Pruning for Robust and Efficient Data-Centric Learning
National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
Entity alignment is to find identical entities in different knowledge graphs (KGs) that refer to the same real-world object. Embedding-based entity alignment techniques have been drawing a lot of attention recently because they can help solve the issue of symbolic heterogeneity in different KGs. However, in this paper, we show that the progress made in the past was due to biased and unchallenging evaluation. We highlight two major flaws in existing datasets that favor embedding-based entity alignment techniques, i.e., the isomorphic graph structures in relation triples and the weak heterogenei…
Université de Lyon（里昂大学）；INSA-Lyon（里昂国立应用科学学院）；Orange Labs（Orange实验室）
The large adoption of the self-attention (i.e. transformer model) and BERT-like training principles has recently resulted in a number of high performing models on a large panoply of vision-and-language problems (such as Visual Question Answering (VQA), image retrieval, etc.). In this paper we claim that these State-Of-The-Art (SOTA) approaches perform reasonably well in structuring information inside a single modality but, despite their impressive performances , they tend to struggle to identify fine-grained inter-modality relationships. Indeed, such relations are frequently assumed to be impl…
Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment
Zhejiang University（浙江大学）；The University of Manchester（曼彻斯特大学）；University of Oxford（牛津大学）；The University of Edinburgh（爱丁堡大学）；Tsinghua University（清华大学）；Donghai laboratory（东海实验室）
Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology（北京智能信息科技重点实验室，计算机科学与技术学院，北京理工大学）；Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University（广东机器感知与智能计算实验室，深圳MSU-BIT大学）；Department of Electrical and Computer System Engineering, Monash University（电子与计算机系统工程系，墨尔本大学）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University at Buffalo（布法罗大学）；Amazon（亚马逊）
Contrastive loss has been increasingly used in learning representations from multiple modalities. In the limit, the nature of the contrastive loss encourages modalities to exactly match each other in the latent space. Yet it remains an open question how the modality alignment affects the downstream task performance. In this paper, based on an information-theoretic argument, we first prove that exact modality alignment is sub-optimal in general for downstream prediction tasks. Hence we advocate that the key of better performance lies in meaningful latent modality structures instead of perfect m…
OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment
Hong Kong University of Science and Technology（香港科学与技术大学）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；UiT – The Arctic University of Norway（挪威北极大学）；University of Copenhagen（哥本哈根大学）；Norwegian Computing Center（挪威计算中心）；University of Massachusetts Amherst（马萨诸塞大学阿姆赫斯特分校）
Université de Sousse（苏塞大学）；Ecole Nationale d’Ingénieurs de Sousse（苏塞国立工程师学院）；Istanbul Technical University（伊斯坦布尔理工大学）
Brain graph synthesis marked a new era for predicting a target brain graph from a source one without incurring the high acquisition cost and processing time of neuroimaging data. However, existing multi-modal graph synthesis frameworks have several limitations. First, they mainly focus on generating graphs from the same domain (intra-modality), overlooking the rich multimodal representations of brain connectivity (inter-modality). Second, they can only handle isomorphic graph generation tasks, limiting their generalizability to synthesizing target graphs with a different node size and topologi…
Canine EEG Helps Human: Cross-Species and Cross-Modality Epileptic Seizure Detection via Multi-Space Alignment
Ministry of Education Key Laboratory of Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（教育部图像处理与智能控制重点实验室，人工智能与自动化学院，华中科技大学）；Hubei Key Laboratory of Brain-inspired Intelligent Systems, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（湖北省脑启发智能系统重点实验室，人工智能与自动化学院，华中科技大学）
Istanbul Technical University（伊斯坦布尔理工大学）；Université de Sousse（苏塞大学）；Ecole Nationale d’Ingénieurs de Sousse（苏塞国立工程师学院）
Synthesizing multimodality medical data provides complementary knowledge and helps doctors make precise clinical decisions. Although promising, existing multimodal brain graph synthesis frameworks have several limitations. First, they mainly tackle only one problem (intra- or inter-modality), limiting their generalizability to synthesizing inter- and intra-modality simultaneously. Second, while few techniques work on super-resolving low-resolution brain graphs within a single modality (i.e., intra), inter-modality graph super-resolution remains unexplored though this would avoid the need for c…
GMAIL: Generative Modality Alignment for generated Image Learning
Department of Machine Learning, CMU, USA（卡内基梅隆大学机器学习系）；Department of Machine Learning, MBZUAI, UAE（马斯克大学人工智能研究所）；Department of Artificial Intelligence, Hanyang University ERICA, South Korea（翰阳大学ERICA人工智能系）
Audio-visual cross-modality knowledge transfer for machine learning-based in-situ monitoring in laser additive manufacturing
McGill University（麦吉尔大学）；Nanyang Technological University（南洋理工大学）；Advanced Remanufacturing and Technology Center (ARTC), Agency of Science, Technology and Research (A*STAR)（新加坡科学技术研究局先进再制造与技术中心）
