The universal model emerges as a promising trend for medical image segmentation, paving up the way to build medical imaging large model (MILM). One popular strategy to build universal models is to encode each task as a one-hot vector and generate dynamic convolutional layers at the end of the decoder to extract the interested target. Although successful, it ignores the correlations among tasks and meanwhile is too late to make the model 'aware' of the ongoing task. To address both issues, we propose a prompt-driven Universal Segmentation model (UniSeg) for multi-task medical image segmentation using diverse modalities and domains. We first devise a learnable universal prompt to describe the correlations among all tasks and then convert this prompt and image features into a task-specific prompt, which is fed to the decoder as a part of its input. Thus, we make the model 'aware' of the ongoing task early and boost the task-specific training of the whole decoder. Our results indicate that the proposed UniSeg outperforms other universal models and single-task models on 11 upstream tasks. Moreover, UniSeg also beats other pre-trained models on two downstream datasets, providing the community with a high-quality pre-trained model for 3D medical image segmentation. Code and model are available at https://github.com/yeerwen/UniSeg.
Zhejiang University（浙江大学）；Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）；Alibaba Group（阿里巴巴集团）
Large-scale pre-trained language models have contributed significantly to natural language processing by demonstrating remarkable abilities as few-shot learners. However, their effectiveness depends mainly on scaling the model parameters and prompt design, hindering their implementation in most real-world applications. This study proposes a novel pluggable, extensible, and efficient approach named DifferentiAble pRompT (DART), which can convert small language models into better few-shot learners without any prompt engineering. The main principle behind this approach involves reformulating pote…
Not Just Object, But State: Compositional Incremental Learning without Forgetting
International School of Information Science & Engineering, Dalian University of Technology（大连理工大学信息科学与工程国际学院）；MAIS&CRIPAC, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
The Chinese University of Hong Kong（香港中文大学）；Shanghai Jiaotong University（上海交通大学）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Science（中国科学院深圳先进技术研究院）
Visual recognition in low-data regimes requires deep neural networks to learn generalized representations from limited training samples. Recently, CLIP-based methods have shown promising few-shot performance benefited from the contrastive language-image pre-training. We then question, if the more diverse pre-training knowledge can be cascaded to further assist few-shot representation learning. In this paper, we propose CaFo, a Cascade of Foundation models that incorporates diverse prior knowledge of various pre-training paradigms for better few-shot learning. Our CaFo incorporates CLIP's langu…
Few-shot Learner Parameterization by Diffusion Time-steps
Nanyang Technological University（南洋理工大学）；Sea AI Lab（Sea AI实验室）；Singapore Management University（新加坡管理大学）；Hefei University of Technology（合肥工业大学）；Skywork AI
The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples, such as domain-specific captioning, question answering, and future event prediction. Existing few-shot video-language learners focus exclusively on the encoder, resulting in the absence of a video-to-text decoder to handle generative tasks. Video captioners have been pretrained on large-scale video-language datasets, but they rely heavily on finetuning and lack the ability to generate text for unseen tasks in a few-shot setting. We propose VidIL, a few-shot Vid…
Leveraging Text-to-Image Diffusion Models for Unsupervised Visual Object Tracking
Information Systems Technology and Design Pillar, Singapore University of Technology and Design（新加坡科技设计大学信息系统技术与设计学院）；State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；Department of Computer Science and Engineering, University at Buffalo, State University of New York（纽约州立大学布法罗分校计算机科学与工程系）；School of Computer Science, Wuhan University（武汉大学计算机学院）
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）
Large-scale multi-modal training with image-text pairs imparts strong generalization to CLIP model. Since training on a similar scale for videos is infeasible, recent approaches focus on the effective transfer of image-based CLIP to the video domain. In this pursuit, new parametric modules are added to learn temporal information and inter-frame relationships which require meticulous design efforts. Furthermore, when the resulting models are learned on videos, they tend to overfit on the given task distribution and lack in generalization aspect. This begs the following question: How to effectiv…
