``Learning to hash'' is a practical solution for efficient retrieval, offering fast search speed and low storage cost. It is widely applied in various applications, such as image-text cross-modal search. In this paper, we explore the potential of enhancing the performance of learning to hash with the proliferation of powerful large pre-trained models, such as Vision-Language Pre-training (VLP) models. We introduce a novel method named Distillation for Cross-Modal Quantization (DCMQ), which leverages the rich semantic knowledge of VLP models to improve hash representation learning. Specifically, we use the VLP as a `teacher' to distill knowledge into a `student' hashing model equipped with codebooks. This process involves the replacement of supervised labels, which are composed of multi-hot vectors and lack semantics, with the rich semantics of VLP. In the end, we apply a transformation termed Normalization with Paired Consistency (NPC) to achieve a discriminative target for distillation. Further, we introduce a new quantization method, Product Quantization with Gumbel (PQG) that promotes balanced codebook learning, thereby improving the retrieval performance. Extensive benchmark testing demonstrates that DCMQ consistently outperforms existing supervised cross-modal hashing approaches, showcasing its significant potential.
BrainAnytime: Anatomy-Aware Cross-Modal Pretraining for Brain Image Analysis with Arbitrary Modality Availability
Department of Biomedical Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China(生物医学工程系,香港理工大学,香港特别行政区,中国);Department of Technology Management for Innovation, The University of Tokyo, Japan(创新技术管理系,东京大学,日本);Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China(数据科学与人工智能系,香港理工大学,香港特别行政区,中国)
InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining
Alibaba Group(阿里巴巴集团);Peking University(北京大学)
用于学习高级多模态表示的多模态预训练是迈向深度学习和人工智能的又一步。在这项工作中,我们提出了一个新的模型,即InterBERT(BERT for Interaction),这是我们的多模态预训练方法M6(MultiModality-to-MultiModality Multitask Mega-transformer)系列的第一个模型。该模型具有很强的建模能力,不同模态的信息流之间的相互作用。单流交互模块能够有效地处理多模态的信息,而上层的双流交互模块保持了各模态的独立性,避免了单模态任务的性能下降。我们使用三个预训练任务对模型进行预训练,包括掩蔽段建模(MSM)、掩蔽区域建模(MRM)和图像-文本匹配(ITM);并在一系列视觉和语言下游任务上对模型进行微调。实验结果表明,InterBERT的性能优于一系列强基线,包括最新的多模态预训练方法,分析表明MSM和MRM是有效的预训练方法,我们的方法在单模态任务中可以达到与BERT相当的性能。此外,我们提出了一个大规模的数据集,用于中文多模态预训练,我们开发了中文InterBERT,这是第一个中文多模态预训练模型。我们在我们提出的来自中国最大的电子商务平台手机淘宝的310万图像-文本对数据集上预训练了中文InterBERT。我们微调的模型,基于文本的图像检索,最近,我们部署了基于主题的推荐模型在线。
Nanjing University of Aeronautics and Astronautics(南京航空航天大学);City University of Hong Kong(香港城市大学);Delft University of Technology(代尔夫特理工大学);The Hong Kong Polytechnic University(香港理工大学);Lingnan University(岭南大学);Hong Kong Metropolitan University(香港都会大学)
本文提出 Cross-BERT 自监督预训练框架,通过点—图像对齐和掩码跨模态建模两项任务挖掘 2D-3D 语义几何关联,提升 3D 点云下游性能。
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation
Intel Labs(英特尔实验室);Microsoft Research Asia(微软亚洲研究院);Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences(中国科学院上海微系统与信息技术研究所)
CromSS: Cross-modal pre-training with noisy labels for remote sensing image segmentation
Technical University of Munich (TUM)(慕尼黑工业大学);Remote Sensing Technology Institute, German Aerospace Center (DLR)(德国航空航天中心遥感技术研究所);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心)
MMATR-X: High-Resolution Optical-SAR Joint Pretraining via Conditioned Alignment and Degraded-Target Prediction
College of Electronic Science and Technology, National University of Defense Technology (NUDT)(电子科学与技术学院,国防科技大学);Beijing Institute of Remote Sensing Information(遥感信息研究所)
Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancement
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University, China(通用人工智能国家重点实验室,智能科学与技术学院,北京大学,中国);Huawei Noah’s Ark Lab, China(华为诺亚方舟实验室,中国)
M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining
Sun Yat-sen University(中山大学);Beijing Jiaotong University(北京交通大学);UiT The Arctic University of Norway(挪威UiT北极大学);PengCheng Laboratory(鹏城实验室);Alibaba Group(阿里巴巴集团);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区)
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
Peking University(北京大学);Tencent Data Platform(腾讯数据平台);Pengcheng Lab(鹏城实验室);National University of Singapore(新加坡国立大学);Nari Technology Development Limited Company(南瑞集团有限公司)
针对视频-语言预训练难以扩展到 N 模态的问题,本文提出 LanguageBind,以语言为纽带通过冻结语言编码器和对比学习对齐多模态,并构建 VIDAL-10M 数据集,在 15 个基准上验证有效性。
Explore the Limits of Omni-modal Pretraining at Scale
The Chinese University of Hong Kong(香港中文大学);University of Chinese Academy of Sciences(中国科学院大学);Chinese Academy of Science(中国科学院);Shanghai AI Laboratory(上海人工智能实验室)
School of Computer Science and Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件学院)
Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework
UMIT TIROL – Private University for Health Sciences and Health Technology(UMIT TIROL – 健康科学与健康技术私立大学);VASCage – Centre on Clinical Stroke Research(VASCage – 临床卒中研究中心);Medical University of Innsbruck(因斯布鲁克医科大学)
Improving Out-of-distribution Human Activity Recognition via IMU-Video Cross-modal Representation Learning
Department of Computer Science & Engineering University of Minnesota(计算机科学与工程系明尼苏达大学);Department of Mechanical Engineering University of Minnesota(机械工程系明尼苏达大学);Department of Neurosurgery University of Minnesota(神经外科系明尼苏达大学)
ProtoCLIP: Prototypical Contrastive Language Image Pretraining
College of Computer and Information, Hohai University(河海大学计算机与信息学院);Science and Technology on Underwater Vehicle Technology Laboratory(水下航行器技术科技重点实验室);MEGVII Research(旷视科技研究院);Nanjing Research Institute of Electronic Engineering(南京电子工程研究所);School of EECS, Peking University(北京大学电子工程与计算机科学学院)
SeLIP: Similarity Enhanced Contrastive Language Image Pretraining for Multi-modal Head MRI
College of Electronic Information and Optical Engineering, Nankai University(南开大学电子信息与光学工程学院);Tianjin Key Laboratory of Optoelectronic Sensor and Sensing Network Technology(天津市光电传感器与传感网络技术重点实验室);Tianjin First Central Hospital, School of Medicine, Nankai University(南开大学医学院附属天津市第一中心医院)
UniEmoX: Cross-modal Semantic-Guided Large-Scale Pretraining for Universal Scene Emotion Perception
School of Artificial Intelligence, Anhui University(安徽大学人工智能学院);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥国家综合科学中心人工智能研究所);Anhui Province Key Laboratory of Affective Computing and Advanced Intelligent Machines, School of Computer Science and Information Engineering, Hefei University of Technology(安徽省情感计算与先进智能机器重点实验室,合肥工业大学计算机科学与信息工程学院);Department of Computer and Network Engineering, The University of Electro-Communications(电子通信大学计算机与网络工程系)