Image-text multimodal representation learning aligns data across modalities and enables important medical applications, e.g., image classification, visual grounding, and cross-modal retrieval. In this work, we establish a connection between multimodal representation learning and multiple instance learning. Based on this connection, we propose a generic framework for constructing permutation-invariant score functions with many existing multimodal representation learning approaches as special cases. Furthermore, we use the framework to derive a novel contrastive learning approach and demonstrate that our method achieves state-of-the-art results in several downstream tasks.
From Semantics, Scene to Instance-awareness: Distilling Foundation Model for Grounded Open-vocabulary Situation Recognition
National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学);Huazhong University of Science and Technology(华中科技大学);The Hong Kong Polytechnic University(香港理工大学)
VectorLLM: Human-like Extraction of Structured Building Contours vis Multimodal LLMs
School of Remote Sensing and Information Engineering(遥感与信息工程学院);College of Oceanography and Space Informatics(海洋学与空间信息学院);School of Surveying and Geoinformation Engineering(测绘与地理信息工程学院)
Pilot: Building the Federated Multimodal Instruction Tuning Framework
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Pengcheng Laboratory(鹏城实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)(中国科学院大学人工智能学院);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
Vrije University of Amsterdam University of Amsterdam Centrum Wiskunde \& Informatic;University of Amsterdam National Institute of Informatics Centrum Wiskunde \& Informatic;Centrum Wiskunde \& Informatic Leiden University University College London;Vrije University of Amsterdam;Centrum Wiskunde \& Informatica Technische Universiteit Delft Guangzhou University;Vrije University of Amsterdam Centrum Wiskunde \& Informatic
MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
Shanghai AI Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);USTC(中国科学技术大学);RIT(罗切斯特理工学院);HIT(哈尔滨工业大学);WHU(武汉大学);MBZUAI(Mohamed bin Zayed 人工智能大学);NUS(新加坡国立大学)
IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection
School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);University of Macau(澳门大学);The University of Hong Kong(香港大学);Inceptio(嬴彻科技);École Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院);Zhejiang University(浙江大学)
Multimodal Hate Detection Using Dual-Stream Graph Neural Networks
Multimodal Intelligence Lab, Department of Computer Science University of Exeter Exeter, UK(埃克塞特大学计算机科学系多模态智能实验室);School of Computer Science University of Birmingham Birmingham, UK(伯明翰大学计算机科学学院)
Hierarchical Prototype-based Domain Priors for Multiple Instance Learning in Multimodal Histopathology Analysis
College of Computer and Information Science, Fujian Agriculture and Forestry University(福建农林大学计算机与信息科学学院);College of Future Technology, Fujian Agriculture and Forestry University(福建农林大学未来技术学院);Digital Fujian Institute of Agricultural Big Data, Fujian Agriculture and Forestry University(福建农业大数据数字福建研究院);Department of Pathology, Clinical Oncology School of Fujian Medical University, and Fujian Cancer Hospital(福建医科大学临床肿瘤学院病理科及福建癌症医院)
PlaneSAM: Multimodal Plane Instance Segmentation Using the Segment Anything Model
School of Computer Science and Information Engineering, Hubei University(湖北大学计算机与信息工程学院);School of Artificial Intelligence, Hubei University(湖北大学人工智能学院);Wuhan University(武汉大学);Institute of Geospatial Intelligence, Wuhan University(武汉大学地理空间智能研究院)
Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos
Columbia University(哥伦比亚大学);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);University of Central Florida(中佛罗里达大学);Goethe University Frankfurt(法兰克福大学);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);IBM Research AI(IBM研究院人工智能部门);UT Austin(德克萨斯大学奥斯汀分校);NYU(纽约大学)
MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusion
Department of Computing, The Hong Kong Polytechnic University(计算系,香港理工大学);Research Institute of Multiple Agents and Embodied Intelligence, Pengcheng Laboratory(多智能体与具身智能研究院,鹏城实验室)
Test-time Adaptive Hierarchical Co-enhanced Denoising Network for Reliable Multimodal Classification
The Guangdong Provincial Key Laboratory of Computational Intelligence and Cyberspace Information, the School of Computer Science and Engineering, South China University of Technology(广东省计算智能与网络信息重点实验室、计算机科学与工程学院、华南理工大学);The Pazhou Laboratory(琶洲实验室)
BuildingView: Constructing Urban Building Exteriors Databases with Street View Imagery and Multimodal Large Language Mode
Thrust of Urban Governance and Design, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)城市治理与设计中心);Spatial Sciences Institute, University of Southern California(南加州大学空间科学学院)
Distribution Aligned Multimodal and Multi-Domain Image Stylization
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Microsoft Research Asia(微软亚洲研究院);Kuaishou Technology(快手科技)