Advances in the field of vision-language contrastive learning have made it possible for many downstream applications to be carried out efficiently and accurately by simply taking the dot product between image and text representations. One of the most representative approaches proposed recently known as CLIP has garnered widespread adoption due to its effectiveness. CLIP is trained with an InfoNCE loss that takes into account both positive and negative samples to help learn a much more robust representation space. This paper reveals that the common downstream practice of taking a dot product is only a zeroth-order approximation of the optimization goal, resulting in a loss of information during test-time. Intuitively, since the model has been optimized based on the InfoNCE loss, test-time procedures should also be in alignment. The question lies in how one can retrieve any semblance of negative samples information during inference in a computationally efficient way. To this end, we propose Distribution Normalization (DN), where we approximate the mean representation of a batch of test samples and use such a mean to represent what would be analogous to negative samples in the InfoNCE loss. DN requires no retraining or fine-tuning and can be effortlessly applied during inference. Extensive experiments on a wide variety of downstream tasks exhibit a clear advantage of DN over the dot product on top of other existing test-time augmentation methods.
Mila（米拉（学习算法研究所））；Université de Montréal（蒙特利尔大学）；ETH Zürich（苏黎世联邦理工学院）；Helmholtz AI（亥姆霍兹人工智能研究所）；Technical University of Munich（慕尼黑工业大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Revisiting Theory of Contrastive Learning for Domain Generalization
Munich Center for Machine Learning（慕尼黑机器学习中心）；Department of Computer Science, Sharif University of Technology（技术学院计算机科学系）；Department of Statistics, LMU Munich（慕尼黑大学统计系）
Knowledge distillation via adaptive instance normalization
Computer Vision Laboratory, University of Nottingham, UK（诺丁汉大学计算机视觉实验室）；Samsung AI Center, Cambridge, UK（三星人工智能中心剑桥）；Queen Mary University of London, London, UK（伦敦玛丽女王大学）
Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings
Physics Department, Sapienza University of Rome（罗马大学物理系）；Institute of Nanotechnology, National Research Council of Italy, CNR-NANOTEC, Rome Unit（意大利国家研究委员会纳米技术研究所，CNR-NANOTEC，罗马分部）
Learning Less Generalizable Patterns with an Asymmetrically Trained Double Classifier for Better Test-Time Adaptation
Univ Lyon（里昂大学）；Ecole Centrale de Lyon（里昂中央理工学院）；CNRS（法国国家科学研究中心）；INSA Lyon（里昂国立应用科学学院）；Univ Claude Bernard Lyon 1（里昂第一大学克洛德·贝尔纳）；Univ Louis Lumière Lyon 2（里昂第二大学路易·卢米埃尔）；Thales LAS France（泰雷兹LAS法国公司）
Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness
University of Science and Technology of China（中国科学技术大学）；Baidu Talent Intelligence Center（百度人才智库中心）；Rutgers, The State University of New Jersey（罗格斯大学新泽西州立大学）
