The pretrain-finetune paradigm has shown outstanding performance on many applications of deep learning, where a model is pre-trained on a upstream large dataset (e.g. ImageNet), and is then fine-tuned to different downstream tasks. Though for most cases, the pre-training stage is conducted based on supervised methods, recent works on self-supervised pre-training have shown powerful transferability and even outperform supervised pre-training on multiple downstream tasks. It thus remains an open question how to better generalize supervised pre-training model to downstream tasks. In this paper, we argue that the worse transferability of existing supervised pre-training methods arise from the negligence of valuable intra-class semantic difference. This is because these methods tend to push images from the same class close to each other despite of the large diversity in their visual contents, a problem to which referred as "overfit of upstream tasks". To alleviate this problem, we propose a new supervised pre-training method based on Leave-One-Out K-Nearest-Neighbor, or LOOK for short. It relieves the problem of overfitting upstream tasks by only requiring each image to share its class label with most of its k nearest neighbors, thus allowing each class to exhibit a multi-mode distribution and consequentially preserving part of intra-class difference for better transferring to downstream tasks. We developed efficient implementation of the proposed method that scales well to large datasets. Experimental studies on multiple downstream tasks show that LOOK outperforms other state-of-the-art methods for supervised and self-supervised pre-training.
What Makes a Good Medical Image Tokenizer? Rethinking Reconstruction and Generation in Medical Image Tokenization
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning, Technical University of Munich（慕尼黑工业大学慕尼黑机器学习中心）；Klinikum rechts der Isar, Technical University of Munich（慕尼黑工业大学伊萨尔河右岸医院）；ELLIS Institute Finland（芬兰ELLIS研究所）；Aalto University（阿尔托大学）；Imperial College London（伦敦帝国理工学院）
Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks
Peking University（北京大学）；Xiaomi EV（小米电动车）；Huazhong University of Science and Technology（华中科技大学）；Beijing Key Laboratory of Data Intelligence and Security (Peking University)（北京数据智能与安全重点实验室（北京大学））；Zhongguancun Academy（中关村学院）
Rethinking Foundation Models for Medical Image Classification through a Benchmark Study on MedMNIST
University of Oxford（牛津大学）；Nuffield Department of Population Health, University of Oxford（牛津大学纳菲尔德人口健康系）；Big Data Institute, University of Oxford（牛津大学大数据研究所）
Rethinking the Bias of Foundation Model under Long-tailed Distribution
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学耿丽人工智能学院）；Beijing Key Laboratory of Big Data Management and Analysis Methods（北京大数据管理与分析方法重点实验室）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；Electrical and Computer Engineering, Carnegie Mellon University（卡内基梅隆大学电气与计算机工程学院）；University of Chinese Academy of Sciences（中国科学院大学）
MaeFuse: Transferring Omni Features with Pretrained Masked Autoencoders for Infrared and Visible Image Fusion via Guided Training
School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；Electronic Information School, Wuhan University（武汉大学电子信息学院）；School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）
Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofing
Great Bay University（大湾区大学）；Nanyang Technological University（南洋理工大学）；University of Oulu（奥卢大学）；Lappeenranta-Lahti University of Technology（拉彭兰塔-拉赫蒂理工大学）；South China University of Technology（华南理工大学）
近年来，基于视觉Transformer（ViT）的多模态学习方法被提出来提高人脸反欺骗（FAS）系统的鲁棒性。然而，仍然没有工作来探索的基本性质（\textit{e.g.}，模态感知输入，合适的多模态预训练，和有效的微调），在香草ViT多模态FAS。在本文中，我们研究了三个关键因素（即，输入、预训练和微调），用于具有RGB、红外（IR）和深度的多模态FAS。首先，在ViT输入方面，我们发现利用局部特征描述符有利于IR模态上的ViT，而不是RGB或深度模态。其次，在观察的效率直接微调整个或部分ViT，我们设计了一个自适应多模态适配器（AMA），它可以有效地聚合本地多模态功能，同时冻结大多数ViT参数。最后，考虑到任务（FAS与通用对象分类）和模态（多模态与单峰）的差距，ImageNet预训练模型对于多模态FAS任务可能是次优的。为了弥合这些差距，我们提出了模态非对称掩码自编码器（M $^{2}$ A $^{2}$ E），用于多模态FAS自监督预训练，而无需昂贵的注释标签。与先前的模态对称自动编码器相比，所提出的M $^{2}$ A $^{2}$ E能够学习更多内在的任务感知表示并且与模态不可知（例如，单峰、双峰和三峰）下游设置。在多模态FAS基准上进行的单峰（RGB，深度，IR）和多模态（RGB+深度，RGB+IR，深度+IR，RGB+深度+IR）设置的广泛实验证明了所提出的方法的优…
