LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
Peking University(北京大学);Tencent Data Platform(腾讯数据平台);Pengcheng Lab(鹏城实验室);National University of Singapore(新加坡国立大学);Nari Technology Development Limited Company(南瑞集团有限公司)
Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, Li Yuan
The video-language (VL) pretraining has achieved remarkable improvement in multiple downstream tasks. However, the current VL pretraining framework is hard to extend to multiple modalities (N modalities, N>=3) beyond vision and language. We thus propose LanguageBind, taking the language as the bind across different modalities because the language modality is well-explored and contains rich semantics. Specifically, we freeze the language encoder acquired by VL pretraining, then train encoders for other modalities with contrastive learning. As a result, all modalities are mapped to a shared feature space, implementing multi-modal semantic alignment. While LanguageBind ensures that we can extend VL modalities to N modalities, we also need a high-quality dataset with alignment data pairs centered on language. We thus propose VIDAL-10M with Video, Infrared, Depth, Audio and their corresponding Language, naming as VIDAL-10M. In our VIDAL-10M, all videos are from short video platforms with complete semantics rather than truncated segments from long videos, and all the video, depth, infrared, and audio modalities are aligned to their textual descriptions. LanguageBind has achieved superior performance on a wide range of 15 benchmarks covering video, audio, depth, and infrared. Moreover, multiple experiments have provided evidence for the effectiveness of LanguageBind in achieving indirect alignment and complementarity among diverse modalities. Code address: https://github.com/PKU-YuanGroup/LanguageBind
Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework
UMIT TIROL – Private University for Health Sciences and Health Technology(UMIT TIROL – 健康科学与健康技术私立大学);VASCage – Centre on Clinical Stroke Research(VASCage – 临床卒中研究中心);Medical University of Innsbruck(因斯布鲁克医科大学)
Multimodal Medical Image Classification via Synergistic Learning Pre-training
College of Biomedical Engineering, Fudan University(复旦大学生物医学工程学院);College of Computer Science and Engineering, Guangxi Normal University(广西师范大学计算机科学与工程学院);Fujian Provincial Key Laboratory of Information Processing and Intelligent Control, School of Computer and Big Data, Minjiang University(福建省信息处理与智能控制重点实验室,闽江学院计算机与大数据学院);Department of Automation Science and Electrical Engineering, Beihang University(北京航空航天大学自动化科学与电气工程学院);Department of Digestive Endoscopy, Fuzhou University Affiliated Provincial Hospital, Provincial Clinical Medical College of Fujian Medical University(福州市大学附属省医院消化内镜科,福建医科大学省临床医学学院);Department of Urology, Fuzhou University Affiliated Provincial Hospital, Provincial Clinical Medical College of Fujian Medical University(福州市大学附属省医院泌尿科,福建医科大学省临床医学学院)
ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining
School of Computer Science(计算机科学学院);Huazhong University of Science and Technology(华中科技大学);Li Auto Inc.(力汽车公司);Institute of AI for Industries, Chinese Academy of Sciences(产业人工智能研究院,中国科学院)
Multimodal Contrastive Pretraining of CBCT and IOS for Enhanced Tooth Segmentation
Department of Computer Science and Engineering(计算机科学与工程系);The Hong Kong University of Science and Technology(香港科学与技术大学);Division of Paediatric Dentistry and Orthodontics, Faculty of Dentistry(牙科学院儿童牙科与正畸科);The University of Hong Kong(香港大学);Delun Dental Hospital(德尔伦牙科医院);Department of Chemical and Biological Engineering(化学与生物工程系);Division of Life Science(生命科学系);HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute(香港科技大学深圳-香港协同创新研究院);State Key Laboratory of Nervous System Disorders(神经系统紊乱国家重点实验室)
BrainAnytime: Anatomy-Aware Cross-Modal Pretraining for Brain Image Analysis with Arbitrary Modality Availability
Department of Biomedical Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China(生物医学工程系,香港理工大学,香港特别行政区,中国);Department of Technology Management for Innovation, The University of Tokyo, Japan(创新技术管理系,东京大学,日本);Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China(数据科学与人工智能系,香港理工大学,香港特别行政区,中国)
Explore the Limits of Omni-modal Pretraining at Scale
The Chinese University of Hong Kong(香港中文大学);University of Chinese Academy of Sciences(中国科学院大学);Chinese Academy of Science(中国科学院);Shanghai AI Laboratory(上海人工智能实验室)
M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining
Sun Yat-sen University(中山大学);Beijing Jiaotong University(北京交通大学);UiT The Arctic University of Norway(挪威UiT北极大学);PengCheng Laboratory(鹏城实验室);Alibaba Group(阿里巴巴集团);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区)
Beijing University of Posts and Telecommunication(北京邮电大学);Rutgers University(罗格斯大学);The Chinese University of Hong Kong(香港中文大学);Chinese Academy of Sciences(中国科学院);Hasso Plattner Institute(哈索·普拉特纳研究院)
Beijing University of Posts and Telecommunications(北京邮电大学);Rutgers University(罗格斯大学);Shanghai AI Laboratory(上海人工智能实验室);Hasso Plattner Institute, University of Potsdam(波茨坦大学哈索普拉特纳研究所);YITU Technology(依图科技)
3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection
School of Electronic, Electrical Engineering and Physics, Fujian University of Technology(福建工程学院电子电气工程学院);School of Computer Science and Engineering, Southeast University, China(东南大学计算机科学与工程学院);Department of Medical Imaging, Suzhou Traditional Chinese Medicine Hospital, China(苏州中医医院影像科)
SIMPLER: H&E-Informed Representation Learning for Structured Illumination Microscopy
Kahlert School of Computing, University of Utah, Salt Lake City, UT 84112, USA Scientific Computing \& Imaging Institute, University of Utah, Salt Lake City, UT 84112, USA Instapath Inc., Houston, TX 77021 Tulane University, Department of Biomedical Engineering, New Orleans, Louisiana, United States
VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Nanjing University of Science and Technology(南京理工大学)
ProtoCLIP: Prototypical Contrastive Language Image Pretraining
College of Computer and Information, Hohai University(河海大学计算机与信息学院);Science and Technology on Underwater Vehicle Technology Laboratory(水下航行器技术科技重点实验室);MEGVII Research(旷视科技研究院);Nanjing Research Institute of Electronic Engineering(南京电子工程研究所);School of EECS, Peking University(北京大学电子工程与计算机科学学院)
LLMs Can Evolve Continually on Modality for X-Modal Reasoning
Dalian University of Technology(大连理工大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Tsinghua University(清华大学);The Hong Kong University of Science and Technology(香港科技大学)
UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalities
Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Khalifa University(哈利法大学);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学);EPFL(洛桑联邦理工学院)