DeepMind；VGG, Dept. of Engineering Science, University of Oxford（牛津大学工程科学系VGG）
Adrià Recasens, Jason Lin, Joāo Carreira, Drew Jaegle, Luyu Wang, Jean-baptiste Alayrac, Pauline Luc, Antoine Miech, Lucas Smaira, Ross Hemsley, Andrew Zisserman
Attention-based models are appealing for multimodal processing because inputs from multiple modalities can be concatenated and fed to a single backbone network - thus requiring very little fusion engineering. The resulting representations are however fully entangled throughout the network, which may not always be desirable: in learning, contrastive audio-visual self-supervised learning requires independent audio and visual features to operate, otherwise learning collapses; in inference, evaluation of audio-visual models should be possible on benchmarks having just audio or just video. In this paper, we introduce Zorro, a technique that uses masks to control how inputs from each modality are routed inside Transformers, keeping some parts of the representation modality-pure. We apply this technique to three popular transformer-based architectures (ViT, Swin and HiP) and show that with contrastive pre-training Zorro achieves state-of-the-art results on most relevant benchmarks for multimodal tasks (AudioSet and VGGSound). Furthermore, the resulting models are able to perform unimodal inference on both video and audio benchmarks such as Kinetics-400 or ESC-50.
Semantics-Guided Multimodal Masked Autoencoder Pretraining for 3D BEV Object Detection
University of Lincoln, Lincoln Centre for Autonomous Systems（林肯大学，林肯自主系统中心）；University of Cambridge, Institute for Manufacturing, Department of Engineering（剑桥大学，制造研究所，工程系）
Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofing
Great Bay University（大湾区大学）；Nanyang Technological University（南洋理工大学）；University of Oulu（奥卢大学）；Lappeenranta-Lahti University of Technology（拉彭兰塔-拉赫蒂理工大学）；South China University of Technology（华南理工大学）
近年来，基于视觉Transformer（ViT）的多模态学习方法被提出来提高人脸反欺骗（FAS）系统的鲁棒性。然而，仍然没有工作来探索的基本性质（\textit{e.g.}，模态感知输入，合适的多模态预训练，和有效的微调），在香草ViT多模态FAS。在本文中，我们研究了三个关键因素（即，输入、预训练和微调），用于具有RGB、红外（IR）和深度的多模态FAS。首先，在ViT输入方面，我们发现利用局部特征描述符有利于IR模态上的ViT，而不是RGB或深度模态。其次，在观察的效率直接微调整个或部分ViT，我们设计了一个自适应多模态适配器（AMA），它可以有效地聚合本地多模态功能，同时冻结大多数ViT参数。最后，考虑到任务（FAS与通用对象分类）和模态（多模态与单峰）的差距，ImageNet预训练模型对于多模态FAS任务可能是次优的。为了弥合这些差距，我们提出了模态非对称掩码自编码器（M $^{2}$ A $^{2}$ E），用于多模态FAS自监督预训练，而无需昂贵的注释标签。与先前的模态对称自动编码器相比，所提出的M $^{2}$ A $^{2}$ E能够学习更多内在的任务感知表示并且与模态不可知（例如，单峰、双峰和三峰）下游设置。在多模态FAS基准上进行的单峰（RGB，深度，IR）和多模态（RGB+深度，RGB+IR，深度+IR，RGB+深度+IR）设置的广泛实验证明了所提出的方法的优…
SELECTOR: Heterogeneous graph network with convolutional masked autoencoder for multimodal robust prediction of cancer survival
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机与通信工程学院）；The Second Xiangya Hospital, Central South University（中南大学湘雅二医院）
Self-control: A Better Conditional Mechanism for Masked Autoregressive Model
Nankai University（南开大学）；School of Statistics and Data Science（统计与数据科学学院）；Key Laboratory of Pure Mathematics and Combinatorics, Ministry of Education（教育部纯数学与组合数学重点实验室）
AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling
Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, China（中国科学院智能信息处理重点实验室（中国科学院计算技术研究所，中国））；University of Chinese Academy of Sciences, China（中国科学院大学）
Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and Voice
Numediart Institute, ISIA Lab, University of Mons（Numediart研究院、ISIA实验室、蒙斯大学）；Institute for Creative Technologies, University of Southern California（创意技术研究所、南加州大学）
3D Masked Autoencoders are Robust Learners of Volumetric and Multimodal Cellular Representations for Microscopy
Institute of AI for Health & Helmholtz AI, Computational Health Center, Helmholtz Munich – German Research Center for Environmental Health（亥姆霍兹慕尼黑中心 - 德国环境健康研究中心，计算健康中心，健康人工智能研究所与亥姆霍兹人工智能）；Department of Medicine III, Ludwig-Maximilian-University Hospital（路德维希-马克西米利安大学医院第三内科）；Department of Physics, Ludwig-Maximilian-University（路德维希-马克西米利安大学物理系）；German Cancer Consortium (DKTK), partner site Munich（德国癌症联盟（DKTK）慕尼黑合作站点）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））
MaskFi: Unsupervised Learning of WiFi and Vision Representations for Multimodal Human Activity Recognition
School of Electrical and Electronics Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与航天工程学院）
Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers
Archimedes, Athena Research Center（阿基米德研究中心）；National Technical University of Athens（希腊国家技术大学）；University of Crete（克里特大学）；IACM-Forth（第四研究机构（IACM-Forth））
Harmonizing Visual Representations for Unified Multimodal Understanding and Generation
S-Lab, Nanyang Technological University（南洋理工大学S实验室）；Shanghai AI Laboratory Research（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；SenseTime Research and Tetras.AI（SenseTime研究部和Tetras.AI）；SenseTime Research（商汤科技研究院）
SCMM: Calibrating Cross-modal Representations for Text-Based Person Search
College of Future Information Technology, Fudan University（复旦大学未来信息技术学院）；Department of Electrical and Computer Engineering, The University of British Columbia（不列颠哥伦比亚大学电气与计算机工程系）；College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）；MEGVII Technology（MEGVII技术）；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；School of AI and CS, Nantong University（南通大学人工智能与计算机科学学院）；Academy of Artificial Intelligence, SMBU（SMBU人工智能学院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）
impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction
Center for Applied AI, Skolkovo Institute of Science and Technology, Moscow, Russian Federation（应用人工智能中心，斯克尔科夫科学与技术研究所，莫斯科，俄罗斯联邦）；BIMAI-Lab, Biomedically Informed Artificial Intelligence Laboratory, University of Sharjah, Sharjah, United Arab Emirates（BIMAI实验室，生物医学导向人工智能实验室，肖贾联合大学，肖贾，阿拉伯联合酋长国）；Ivannikov Institute for System Programming of the Russian Academy of Sciences, Moscow, Russian Federation（伊万诺夫俄罗斯科学院系统编程研究所，莫斯科，俄罗斯联邦）
UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modeling
Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（高性能计算研究所（A*STAR））；Singapore Eye Research Institute, Singapore National Eye Centre（新加坡眼科研究所，新加坡国家眼科中心）
Vehicle-centric Perception via Multimodal Structured Pre-training
Information Materials and Intelligent Sensing Laboratory of Anhui Province（安徽省信息材料与智能感知实验室）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）；the School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
