This paper aims to establish a generic multi-modal foundation model that has the scalable capability to massive downstream applications in E-commerce. Recently, large-scale vision-language pretraining approaches have achieved remarkable advances in the general domain. However, due to the significant differences between natural and product images, directly applying these frameworks for modeling image-level representations to E-commerce will be inevitably sub-optimal. To this end, we propose an instance-centric multi-modal pretraining paradigm called ECLIP in this work. In detail, we craft a decoder architecture that introduces a set of learnable instance queries to explicitly aggregate instance-level semantics. Moreover, to enable the model to focus on the desired product instance without reliance on expensive manual annotations, two specially configured pretext tasks are further proposed. Pretrained on the 100 million E-commerce-related data, ECLIP successfully extracts more generic, semantic-rich, and robust representations. Extensive experimental results show that, without further fine-tuning, ECLIP surpasses existing methods by a large margin on a broad range of downstream tasks, demonstrating the strong transferability to real-world E-commerce applications.
BrainAnytime: Anatomy-Aware Cross-Modal Pretraining for Brain Image Analysis with Arbitrary Modality Availability
Department of Biomedical Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, China(生物医学工程系,香港理工大学,香港特别行政区,中国);Department of Technology Management for Innovation, The University of Tokyo, Japan(创新技术管理系,东京大学,日本);Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China(数据科学与人工智能系,香港理工大学,香港特别行政区,中国)
Nanjing University of Aeronautics and Astronautics(南京航空航天大学);City University of Hong Kong(香港城市大学);Delft University of Technology(代尔夫特理工大学);The Hong Kong Polytechnic University(香港理工大学);Lingnan University(岭南大学);Hong Kong Metropolitan University(香港都会大学)
本文提出 Cross-BERT 自监督预训练框架,通过点—图像对齐和掩码跨模态建模两项任务挖掘 2D-3D 语义几何关联,提升 3D 点云下游性能。
InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining
Alibaba Group(阿里巴巴集团);Peking University(北京大学)
用于学习高级多模态表示的多模态预训练是迈向深度学习和人工智能的又一步。在这项工作中,我们提出了一个新的模型,即InterBERT(BERT for Interaction),这是我们的多模态预训练方法M6(MultiModality-to-MultiModality Multitask Mega-transformer)系列的第一个模型。该模型具有很强的建模能力,不同模态的信息流之间的相互作用。单流交互模块能够有效地处理多模态的信息,而上层的双流交互模块保持了各模态的独立性,避免了单模态任务的性能下降。我们使用三个预训练任务对模型进行预训练,包括掩蔽段建模(MSM)、掩蔽区域建模(MRM)和图像-文本匹配(ITM);并在一系列视觉和语言下游任务上对模型进行微调。实验结果表明,InterBERT的性能优于一系列强基线,包括最新的多模态预训练方法,分析表明MSM和MRM是有效的预训练方法,我们的方法在单模态任务中可以达到与BERT相当的性能。此外,我们提出了一个大规模的数据集,用于中文多模态预训练,我们开发了中文InterBERT,这是第一个中文多模态预训练模型。我们在我们提出的来自中国最大的电子商务平台手机淘宝的310万图像-文本对数据集上预训练了中文InterBERT。我们微调的模型,基于文本的图像检索,最近,我们部署了基于主题的推荐模型在线。
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation
Intel Labs(英特尔实验室);Microsoft Research Asia(微软亚洲研究院);Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences(中国科学院上海微系统与信息技术研究所)
CromSS: Cross-modal pre-training with noisy labels for remote sensing image segmentation
Technical University of Munich (TUM)(慕尼黑工业大学);Remote Sensing Technology Institute, German Aerospace Center (DLR)(德国航空航天中心遥感技术研究所);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心)
MMATR-X: High-Resolution Optical-SAR Joint Pretraining via Conditioned Alignment and Degraded-Target Prediction
College of Electronic Science and Technology, National University of Defense Technology (NUDT)(电子科学与技术学院,国防科技大学);Beijing Institute of Remote Sensing Information(遥感信息研究所)
M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining
Sun Yat-sen University(中山大学);Beijing Jiaotong University(北京交通大学);UiT The Arctic University of Norway(挪威UiT北极大学);PengCheng Laboratory(鹏城实验室);Alibaba Group(阿里巴巴集团);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区)
Explore the Limits of Omni-modal Pretraining at Scale
The Chinese University of Hong Kong(香港中文大学);University of Chinese Academy of Sciences(中国科学院大学);Chinese Academy of Science(中国科学院);Shanghai AI Laboratory(上海人工智能实验室)
School of Computer Science and Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件学院)
SIMPLER: H&E-Informed Representation Learning for Structured Illumination Microscopy
Kahlert School of Computing, University of Utah, Salt Lake City, UT 84112, USA Scientific Computing \& Imaging Institute, University of Utah, Salt Lake City, UT 84112, USA Instapath Inc., Houston, TX 77021 Tulane University, Department of Biomedical Engineering, New Orleans, Louisiana, United States
Multi-view Video-Pose Pretraining for Operating Room Surgical Activity Recognition
University of Strasbourg(斯特拉斯堡大学);CNRS(法国国家科学研究中心);INSERM(法国国家健康与医学研究院);IHU Strasbourg(斯特拉斯堡大学医院研究所);Intuitive Surgical Inc.(直觉外科公司);University Hospital of Strasbourg(斯特拉斯堡大学医院)
Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework
UMIT TIROL – Private University for Health Sciences and Health Technology(UMIT TIROL – 健康科学与健康技术私立大学);VASCage – Centre on Clinical Stroke Research(VASCage – 临床卒中研究中心);Medical University of Innsbruck(因斯布鲁克医科大学)
Improving Out-of-distribution Human Activity Recognition via IMU-Video Cross-modal Representation Learning
Department of Computer Science & Engineering University of Minnesota(计算机科学与工程系明尼苏达大学);Department of Mechanical Engineering University of Minnesota(机械工程系明尼苏达大学);Department of Neurosurgery University of Minnesota(神经外科系明尼苏达大学)
ProtoCLIP: Prototypical Contrastive Language Image Pretraining
College of Computer and Information, Hohai University(河海大学计算机与信息学院);Science and Technology on Underwater Vehicle Technology Laboratory(水下航行器技术科技重点实验室);MEGVII Research(旷视科技研究院);Nanjing Research Institute of Electronic Engineering(南京电子工程研究所);School of EECS, Peking University(北京大学电子工程与计算机科学学院)
Beijing University of Posts and Telecommunication(北京邮电大学);Rutgers University(罗格斯大学);The Chinese University of Hong Kong(香港中文大学);Chinese Academy of Sciences(中国科学院);Hasso Plattner Institute(哈索·普拉特纳研究院)
SeLIP: Similarity Enhanced Contrastive Language Image Pretraining for Multi-modal Head MRI
College of Electronic Information and Optical Engineering, Nankai University(南开大学电子信息与光学工程学院);Tianjin Key Laboratory of Optoelectronic Sensor and Sensing Network Technology(天津市光电传感器与传感网络技术重点实验室);Tianjin First Central Hospital, School of Medicine, Nankai University(南开大学医学院附属天津市第一中心医院)
UniEmoX: Cross-modal Semantic-Guided Large-Scale Pretraining for Universal Scene Emotion Perception
School of Artificial Intelligence, Anhui University(安徽大学人工智能学院);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥国家综合科学中心人工智能研究所);Anhui Province Key Laboratory of Affective Computing and Advanced Intelligent Machines, School of Computer Science and Information Engineering, Hefei University of Technology(安徽省情感计算与先进智能机器重点实验室,合肥工业大学计算机科学与信息工程学院);Department of Computer and Network Engineering, The University of Electro-Communications(电子通信大学计算机与网络工程系)