Performance of convolutional neural networks (CNNs) in image analysis tasks is often marred in the presence of acquisition-related distribution shifts between training and test images. Recently, it has been proposed to tackle this problem by fine-tuning trained CNNs for each test image. Such test-time-adaptation (TTA) is a promising and practical strategy for improving robustness to distribution shifts as it requires neither data sharing between institutions nor annotating additional data. Previous TTA methods use a helper model to increase similarity between outputs and/or features extracted from a test image with those of the training images. Such helpers, which are typically modeled using CNNs, can be task-specific and themselves vulnerable to distribution shifts in their inputs. To overcome these problems, we propose to carry out TTA by matching the feature distributions of test and training images, as modelled by a field-of-experts (FoE) prior. FoEs model complicated probability distributions as products of many simpler expert distributions. We use 1D marginal distributions of a trained task CNN's features as experts in the FoE model. Further, we compute principal components of patches of the task CNN's features, and consider the distributions of PCA loadings as additional experts. We validate the method on 5 MRI segmentation tasks (healthy tissues in 4 anatomical regions and lesions in 1 one anatomy), using data from 17 clinics, and on a MRI registration task, using data from 3 clinics. We find that the proposed FoE-based TTA is generically applicable in multiple tasks, and outperforms all previous TTA methods for lesion segmentation. For healthy tissue segmentation, the proposed method outperforms other task-agnostic methods, but a previous TTA method which is specifically designed for segmentation performs the best for most of the tested datasets. Our code is publicly available.
SAGE: Shape-Adapting Gated Experts for Adaptive Histopathology Image Segmentation
University of Science, VNU-HCM（越南国家大学科学学院）；Trivita AI；University of Technology, VNU-HCM（越南国家大学技术学院）；Michigan State University, USA（美国密歇根州立大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Concordia University（康考迪亚大学）；University of Toronto（多伦多大学）
In this paper, we tackle the problem of domain shift. Most existing methods perform training on multiple source domains using a single model, and the same trained model is used on all unseen target domains. Such solutions are sub-optimal as each target domain exhibits its own specialty, which is not adapted. Furthermore, expecting single-model training to learn extensive knowledge from multiple source domains is counterintuitive. The model is more biased toward learning only domain-invariant features and may result in negative knowledge transfer. In this work, we propose a novel framework for …
Controllable-LPMoE: Adapting to Challenging Object Segmentation via Dynamic Local Priors from Mixture-of-Experts
PCA Lab, Nanjing University of Science and Technology, Nanjing, China（南京理工大学科学与工程学院实验室）；PCA Lab, VCIP, College of Computer Science, Nankai University, Tianjin, China（南开大学计算机学院）
Carnegie Mellon University（卡内基梅隆大学）；Waseda University（早稻田大学）；Yahoo Japan Corporation（雅虎日本公司）
Question categorization and expert retrieval methods have been crucial for information organization and accessibility in community question & answering (CQA) platforms. Research in this area, however, has dealt with only the text modality. With the increasing multimodal nature of web content, we focus on extending these methods for CQA questions accompanied by images. Specifically, we leverage the success of representation learning for text and images in the visual question answering (VQA) domain, and adapt the underlying concept and architecture for automated category classification and exper…
Training-free image style alignment for self-adapting domain shift on handheld ultrasound devices
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai Advanced Research Institute, Chinese Academy of Sciences（中国科学院上海高等研究院）；University of Chinese Academy of Sciences（中国科学院大学）；College of Computer Science, Sichuan University（四川大学计算机学院）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；The Chinese University of Hong Kong（香港中文大学）；Beth Israel Deaconess Medical Center, Harvard Medical School（哈佛医学院贝斯以色列女执事医疗中心）；Institute of High Performance Computing, Agency for Science, Technology and Research（新加坡科技研究局高性能计算研究所）
Fast Omni-Directional Image Super-Resolution: Adapting the Implicit Image Function with Pixel and Semantic-Wise Spherical Geometric Priors
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy(SZ)（广东省人工智能与数字经济实验室（深圳））；Peng Cheng Laboratory（鹏城实验室）
Adapting Dense Vision-Language Relationships for Multi-label Classification with Partial Label
State Key Laboratory of Virtual Reality Technology and Systems（虚拟现实技术与系统国家重点实验室）；School of Computer Science and Engineering（计算机科学与工程学院）；Beihang University（北京航空航天大学）
Adapting Vision Foundation Models with Cascaded Semantics
University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；Carnegie Mellon University（卡内基梅隆大学）；University of Missouri–Kansas City（密苏里大学堪萨斯城分校）；Northeastern University（东北大学）；Tulane University（杜兰大学）；University of Bristol（布里斯托大学）；Oak Ridge National Laboratory（橡树岭国家实验室）
With the development of deep learning, the single super-resolution image reconstruction network models are becoming more and more complex. Small changes in hyperparameters of the models have a greater impact on model performance. In the existing works, experts have gradually explored a set of optimal model parameters based on empirical values or performing brute-force search. In this paper, we introduce a new super-resolution image reconstruction generative adversarial network framework, and a Bayesian optimization method used to optimizing the hyperparameters of the generator and discriminato…
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
Detecting action units (AUs) on human faces is challenging because various AUs make subtle facial appearance change over various regions at different scales. Current works have attempted to recognize AUs by emphasizing important regions. However, the incorporation of expert prior knowledge into region definition remains under-exploited, and current AU detection approaches do not use regional convolutional neural networks (R-CNN) with expert prior knowledge to directly focus on AU-related regions adaptively. By incorporating expert prior knowledge, we propose a novel R-CNN based model named AU …
Adapting Large VLMs with Iterative and Manual Instructions for Generative Low-light Enhancement
Dalian University of Technology（大连理工大学）；National University of Singapore（新加坡国立大学）；Hong Kong Polytechnic University（香港理工大学）；University of California, San Francisco（加州大学旧金山分校）；Nanjing University of Science and Technology（南京理工大学）
We present a tree-structured network architecture for large scale image classification. The trunk of the network contains convolutional layers optimized over all classes. At a given depth, the trunk splits into separate branches, each dedicated to discriminate a different subset of classes. Each branch acts as an expert classifying a set of categories that are difficult to tell apart, while the trunk provides common knowledge to all experts in the form of shared features. The training of our "network of experts" is completely end-to-end: the partition of categories into disjoint subsets is lea…
Guiding the Experts: Semantic Priors for Efficient and Focused MoE Routing
Department of Computer Science（计算机科学系）；Beijing Jiaotong University（北京交通大学）；Kuaishou Technology（快手科技）；AImageLab；Dipartimento di Ingegneria（工程系）；University of Modena and Reggio Emilia（莫德纳和雷吉奥艾米莉亚大学）
MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentation
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University（国家多媒体软件工程研究中心，计算机学院，武汉大学）；Hong Kong University of Science and Technology（香港科学与技术大学）；Horizon Robotics
CoRE-UIR: Prior-guided common and residual experts for efficient all-in-one remote sensing image restoration
School of Geodesy and Geomatics, Wuhan University（武汉大学测绘学院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）
PSReg: Prior-guided Sparse Mixture of Experts for Point Cloud Registration
Shanghai Jiao Tong University（上海交通大学）；Jiangxi University of Finance and Economics（江西财经大学）；Shandong University（山东大学）；Anqing Normal University（安庆师范大学）；Huaihua University（怀化学院）
Several recent papers use image denoising with a Fields of Experts prior to benchmark discrete optimization methods. We show that a non-linear least squares solver significantly outperforms all known discrete methods on this problem.
Scalpel-SAM: A Semi-Supervised Paradigm for Adapting SAM to Infrared Small Object Detection
School of Computer Science, Chengdu University（成都大学计算机科学学院）；China University of Geosciences（中国地质大学）；Zhejiang University（浙江大学）；Intelligent Computing Infrastructure Innovation Center, Zhejiang Lab（浙江实验室智能计算基础设施创新中心）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
This work presents a method for adapting a single, fixed deep neural network to multiple tasks without affecting performance on already learned tasks. By building upon ideas from network quantization and pruning, we learn binary masks that piggyback on an existing network, or are applied to unmodified weights of that network to provide good performance on a new task. These masks are learned in an end-to-end differentiable fashion, and incur a low overhead of 1 bit per network parameter, per task. Even though the underlying network is fixed, the ability to mask individual weights allows for the…
South China University of Technology（华南理工大学）；Tencent AI Lab（腾讯人工智能实验室）；National University of Singapore（新加坡国立大学）；Northwestern Polytechnical University（西北工业大学）；Pazhou Laboratory（琶洲实验室）
In real-world applications, data often come in a growing manner, where the data volume and the number of classes may increase dynamically. This will bring a critical challenge for learning: given the increasing data volume or the number of classes, one has to instantaneously adjust the neural model capacity to obtain promising performance. Existing methods either ignore the growing nature of data or seek to independently search an optimal architecture for a given dataset, and thus are incapable of promptly adjusting the architectures for the changed data. To address this, we present a neural a…
While 3D GANs have recently demonstrated the high-quality synthesis of multi-view consistent images and 3D shapes, they are mainly restricted to photo-realistic human portraits. This paper aims to extend 3D GANs to a different, but meaningful visual form: artistic portrait drawings. However, extending existing 3D GANs to drawings is challenging due to the inevitable geometric ambiguity present in drawings. To tackle this, we present Dr.3D, a novel adaptation approach that adapts an existing 3D GAN to artistic drawings. Dr.3D is equipped with three novel components to handle the geometric ambig…
