Test-time adaptation (TTA) aims to align a model to shifting test domains using only unlabeled streaming data. Most existing methods implicitly infer a single global domain distribution, ignoring the multidimensional and sample-specific nature of real-world domain shifts, leading to fragile adaptation. We propose DOME, an effective domain encoder that explicitly models each sample's domain in a zero-shot manner. DOME leverages vision-language pretraining to extract dense, continuous representations, parameterizes domains as distributional variables, and introduces a momentum-updated sparse domain bank for disentangled supervision. By injecting these explicit domain cues into downstream models, even a basic entropy-minimization TTA strategy achieves state-of-the-art performance across ImageNet-C, ImageNet-R, and ImageNet-Sketch, outperforming complex TTA approaches. Our results demonstrate that robust adaptation stems not from intricate adaptation algorithms, but from explicit, structured domain representation.
University of Maryland, Baltimore County（马里兰大学巴尔的摩郡分校）；Facebook AI（Facebook 人工智能研究院）；UC Davis（加州大学戴维斯分校）
Using natural language as a supervision for training visual recognition models holds great promise. Recent works have shown that if such supervision is used in the form of alignment between images and captions in large training datasets, then the resulting aligned models perform well on zero-shot classification as downstream tasks2. In this paper, we focus on teasing out what parts of the language supervision are essential for training zero-shot image classification models. Through extensive and careful experiments, we show that: 1) A simple Bag-of-Words (BoW) caption could be used as a replac…
University of Science and Technology of China（中国科学技术大学）
Generalized Zero-Shot Learning (GZSL) targets recognizing new categories by learning transferable image representations. Existing methods find that, by aligning image representations with corresponding semantic labels, the semantic-aligned representations can be transferred to unseen categories. However, supervised by only seen category labels, the learned semantic knowledge is highly task-specific, which makes image representations biased towards seen categories. In this paper, we propose a novel Dual-Contrastive Embedding Network (DCEN) that simultaneously learns task-specific and task-indep…
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Youedata Co., Ltd.（优数数据有限公司）；University of Electronic Science and Technology of China（电子科技大学）；University of Science and Technology Beijing（北京科技大学）
Action recognition in videos has attracted a lot of attention in the past decade. In order to learn robust models, previous methods usually assume videos are trimmed as short sequences and require ground-truth annotations of each video frame/sequence, which is quite costly and time-consuming. In this paper, given only video-level annotations, we propose a novel weakly supervised framework to simultaneously locate action frames as well as recognize actions in untrimmed videos. Our proposed framework consists of two major components. First, for action frame localization, we take advantage of the…
State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (i…
Microsoft（微软）；University of California Berkeley（加州大学伯克利分校）
The new generation of state-of-the-art computer vision systems are trained from natural language supervision, ranging from simple object category names to descriptive captions. This form of supervision ensures high generality and usability of the learned visual models, due to the broad concept coverage achieved via large-scale data collection process. Alternatively, we argue that learning with external knowledge is a promising way which leverages a much more structured source of supervision and offers sample efficiency. We propose K-LITE, a simple strategy to leverage external knowledge for bu…
Seoul National University（首尔国立大学）；V.DO Inc（V.DO公司）
In this work, we introduce a novel weakly supervised object detection (WSOD) paradigm to detect objects belonging to rare classes that have not many examples using transferable knowledge from human-object interactions (HOI). While WSOD shows lower performance than full supervision, we mainly focus on HOI as the main context which can strongly supervise complex semantics in images. Therefore, we propose a novel module called RRPN (relational region proposal network) which outputs an object-localizing attention map only with human poses and action verbs. In the source domain, we fully train an o…
We present a technique to improve the transferability of deep representations learned on small labeled datasets by introducing self-supervised tasks as auxiliary loss functions. While recent approaches for self-supervised learning have shown the benefits of training on large unlabeled datasets, we find improvements in generalization even on small datasets and when combined with strong supervision. Learning representations with self-supervised losses reduces the relative error rate of a state-of-the-art meta-learner by 5-25% on several few-shot learning benchmarks, as well as off-the-shelf deep…
University of Southern California（南加利福尼亚大学）；Google Research（谷歌研究院）
We study how to transfer representations pretrained on source tasks to target tasks in visual percept based RL. We analyze two popular approaches: freezing or finetuning the pretrained representations. Empirical studies on a set of popular tasks reveal several properties of pretrained representations. First, finetuning is required even when pretrained representations perfectly capture the information required to solve the target task. Second, finetuned representations improve learnability and are more robust to noise. Third, pretrained bottom layers are task-agnostic and readily transferable t…
The University of Hong Kong（香港大学）；Xiamen University（厦门大学）
Pre-training lays the foundation for recent successes in radiograph analysis supported by deep learning. It learns transferable image representations by conducting large-scale fully-supervised or self-supervised learning on a source domain. However, supervised pre-training requires a complex and labor intensive two-stage human-assisted annotation process while self-supervised learning cannot compete with the supervised paradigm. To tackle these issues, we propose a cross-supervised methodology named REviewing FreE-text Reports for Supervision (REFERS), which acquires free supervision signals f…
Meta AI；Cranberry-Lemon University（蔓越莓柠檬大学）；University of the Witwatersrand（威特沃特斯兰德大学）
Costly, noisy, and over-specialized, labels are to be set aside in favor of unsupervised learning if we hope to learn cheap, reliable, and transferable models. To that end, spectral embedding, self-supervised learning, or generative modeling have offered competitive solutions. Those methods however come with numerous challenges \textit{e.g.} estimating geodesic distances, specifying projector architectures and anti-collapse losses, or specifying decoder architectures and reconstruction losses. In contrast, we introduce a simple explainable alternative -- coined \textbf{DIET} -- to learn repres…
ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision
IBM Research Zurich, Zurich, Switzerland（IBM苏黎世研究实验室，瑞士苏黎世）；ETH Zurich, Computer Vision（苏黎世联邦理工学院，计算机视觉）；ETH AI Center, Switzerland（苏黎世联邦理工学院人工智能中心，瑞士）；ETH Zurich, Photogrammetry（苏黎世联邦理工学院，摄影测量学）；Microsoft, Switzerland（微软公司，瑞士）
Looking Beyond the Scale: Do Surgical Skill Models Learn Transferable Representations Across Assessment Rubrics?
National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；DKFZ（德国癌症研究中心）；Faculty of Medicine and University Hospital Carl Gustav Carus（卡尔·古斯塔夫·卡鲁斯医学院及大学医院）；TUD Dresden University of Technology（德累斯顿工业大学）；Helmholtz-Zentrum Dresden-Rossendorf (HZDR)（德累斯顿-罗森多夫亥姆霍兹中心）；BMFTR Research Hub 6G-Life（BMFTR 6G生活研究中心）；Deutsches Krebsforschungszentrum (DKFZ)（德国癌症研究中心）；The Centre for Tactile Internet with Human-in-the-Loop (CeTI)（人在回路触觉互联网中心）
Amortised Invariance Learning for Contrastive Self-Supervision
University of Edinburgh（爱丁堡大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Heidelberg Institute for Theoretical Studies（海德堡理论研究所）；Samsung AI Research Centre（三星人工智能研究中心）
The development of CLIP [Radford et al., 2021] has sparked a debate on whether language supervision can result in vision models with more transferable representations than traditional image-only methods. Our work studies this question through a carefully controlled comparison of two approaches in terms of their ability to learn representations that generalize to downstream classification tasks. We find that when the pre-training dataset meets certain criteria -- it is sufficiently large and contains descriptive captions with low variability -- image-only methods do not match CLIP's transfer pe…
Flood disasters cause enormous social and economic losses. However, both traditional physical models and learning-based flood forecasting models require massive historical flood data to train the model parameters. When come to some new site that does not have sufficient historical data, the model performance will drop dramatically due to overfitting. This technical report presents a Flood Domain Adaptation Network (FloodDAN), a baseline of applying Unsupervised Domain Adaptation (UDA) to the flood forecasting problem. Specifically, training of FloodDAN includes two stages: in the first stage, …
See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRI
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能全国重点实验室）；National Engineering Research Center of Visual Information and Applications（视觉信息与应用国家工程研究中心）；Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Improving the generalization of gait recognition with limited datasets
School of Computer Science, Wuhan University（武汉大学计算机学院）；Department of Automation, School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学自动化系）；Shenzhen Institute of Advanced Study, University of Electronic Science and Technology of China (UESTC)（电子科技大学深圳研究院）；Department of Computer Science and Engineering, Southern University of Science and Technology (SUSTech)（南方科技大学计算机科学与工程系）
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Chinese PLA General Hospital（中国人民解放军总医院）；University of Arkansas at Little Rock（阿肯色大学小石城分校）；Indiana University-Purdue University Indianapolis（印第安纳大学-普渡大学印第安纳波利斯分校）
Weakly-supervised learning has attracted growing research attention on medical lesions segmentation due to significant saving in pixel-level annotation cost. However, 1) most existing methods require effective prior and constraints to explore the intrinsic lesions characterization, which only generates incorrect and rough prediction; 2) they neglect the underlying semantic dependencies among weakly-labeled target enteroscopy diseases and fully-annotated source gastroscope lesions, while forcefully utilizing untransferable dependencies leads to the negative performance. To tackle above issues, …
Pretraining convolutional neural networks via self-supervision, and applying them in transfer learning, is an incredibly fast-growing field that is rapidly and iteratively improving performance across practically all image domains. Meanwhile, model ensembling is one of the most universally applicable techniques in supervised learning literature and practice, offering a simple solution to reliably improve performance. But how to optimally combine self-supervised models to maximize representation quality has largely remained unaddressed. In this work, we provide a framework to perform self-super…
Xpeng motors（小鹏汽车）；Nanyang Technological University（南洋理工大学）；University of Pennsylvania（宾夕法尼亚大学）；Microsoft（微软）
Unsupervised Domain Adaptation (UDA), a branch of transfer learning where labels for target samples are unavailable, has been widely researched and developed in recent years with the help of adversarially trained models. Although existing UDA algorithms are able to guide neural networks to extract transferable and discriminative features, classifiers are merely trained under the supervision of labeled source data. Given the inevitable discrepancy between source and target domains, the classifiers can hardly be aware of the target classification boundaries. In this paper, Shuffle Augmentation o…
Accurate segmentation of organelle instances, e.g., mitochondria, is essential for electron microscopy analysis. Despite the outstanding performance of fully supervised methods, they highly rely on sufficient per-pixel annotated data and are sensitive to domain shift. Aiming to develop a highly annotation-efficient approach with competitive performance, we focus on weakly-supervised domain adaptation (WDA) with a type of extremely sparse and weak annotation demanding minimal annotation efforts, i.e., sparse point annotations on only a small subset of object instances. To reduce performance deg…
