Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(西安交通大学人工智能与机器人研究所);School of Electrical Engineering, Xi’an University of Technology(西安理工大学电气工程学院)
Large pre-trained vision language models (VLMs) have shown impressive zero-shot ability on downstream tasks with manually designed prompt. To further adapt VLMs to downstream tasks, soft prompt is proposed to replace manually designed prompt, which undergoes fine-tuning based on specific domain data. Prior prompt learning methods primarily learn a fixed prompt or residuled prompt from training samples. However, the learned prompts lack diversity and ignore information about unseen domains. In this paper, we reframe the prompt learning framework from a generative perspective and propose a simple yet efficient method for the Domain Generalization (DG) task, namely Soft Prompt Generation (SPG). Specifically, SPG consists of a two-stage training phase and an inference phase. During the training phase, we introduce soft prompt label for each domain, aiming to incorporate the generative model domain knowledge. During the inference phase, the generator of the generative model is employed to obtain instance-specific soft prompts for the unseen target domain. Extensive experiments on five domain generalization benchmarks of three DG tasks demonstrate that SPG achieves state-of-the-art performance. The code is available at https://github.com/renytek13/Soft-Prompt-Generation-with-CGAN.
BioVLM: Routing Prompts, Not Parameters, for Cross-Modality Generalization in Biomedical VLMs
University of Trento(特伦托大学);Carnegie Mellon University(卡内基梅隆大学);LNMIIT Jaipur(贾伊普尔 LNMIIT);MBZUAI;Sunandan Divatia School of Science(Sunandan Divatia 科学学院);IIT Bombay(博伊斯大学)
Concept-Guided Prompt Learning for Generalization in Vision-Language Models
Harbin Institute of Technology(哈尔滨工业大学);Southern University of Science and Technology(南方科技大学);Carnegie Mellon University(卡内基梅隆大学);Pengcheng Laboratory(鹏城实验室)
Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Models
Hangzhou Dianzi University(杭州电子科技大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);The Hong Kong Polytechnic University(香港理工大学);Macao Polytechnic University(澳门理工学院);Zhejiang Provincial Key Laboratory of Low Altitude Ubiquitous Networking Technology, HDU(浙江省低空 ubiquitous 网络技术重点实验室,HDU)
CoDoL: Conditional Domain Prompt Learning for Out-of-Distribution Generalization
East China Normal University(东华师范大学);Xidian University(西安电子科技大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));The University of Oxford(牛津大学);Tsinghua University(清华大学)
Department of Computer Science and Engineering, Southern University of Science and Technology(南方科技大学计算机科学与工程系);Department of Computer Science and Engineering, Hong Kong University of Science and Technology(香港科技大学计算机科学与工程系);School of Information and Control Engineering, China University of Mining and Technology(中国矿业大学信息与控制工程学院)
Prompt Tuning with Soft Context Sharing for Vision-Language Models
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室);Engineering Laboratory for Intelligent Industrial Vision, CASIA(中国科学院自动化研究所智能工业视觉工程实验室);Research Center of Aerospace Information, CASIA(中国科学院自动化研究所航天信息研究中心)
A Similarity Paradigm Through Textual Regularization Without Forgetting
Shanghai Jiao Tong University(上海交通大学);Hong Kong Baptist University(香港浸会大学);Northeastern University(东北大学);University of Science and Technology of China(中国科学技术大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
Federated Domain Generalization via Prompt Learning and Aggregation
School of Computing and Artificial Intelligence, Shandong University of Finance and Economics(山东财经大学计算与人工智能学院);School of Computer Science and Technology, Shandong Jianzhu University(山东建筑大学计算机科学与技术学院);College of Electronics and Information Engineering, Tongji University(同济大学电子与信息工程学院)
本文提出 PLAN 框架,通过提示学习与聚合在联邦域泛化场景中适配视觉-语言模型,利用本地和全局提示的安全交换与聚合提升未见域泛化能力,同时显著降低计算与通信开销。
MoPD: Mixture-of-Prompts Distillation for Vision-Language Models
Department of Computer Science and Engineering, Southern University of Science and Technology(计算机科学与工程系,南方科技大学);School of Mathematical and Computer Sciences, University of Adelaide(数学与计算机科学学院,阿德莱德大学)
PromptIQA: Boosting the Performance and Generalization for No-Reference Image Quality Assessment via Prompts
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);OPPO(OPPO公司)
Prompt-Driven Dynamic Object-Centric Learning for Single Domain Generalization
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);School of Electronic Engineering, Xidian University(西安电子科技大学电子工程学院);Peng Cheng Laboratory(鹏城实验室)
From Adaptation to Generalization: Adaptive Visual Prompting for Medical Image Segmentation
Technical University of Munich(慕尼黑技术大学);Korea University(韩国大学);Kyung Hee University(庆熙大学);Seoul National University of Science and Technology(首尔科学技术大学)
Prompt-Free Lightweight SAM Adaptation for Histopathology Nuclei Segmentation with Strong Cross-Dataset Generalization
School of Information and Communication Technology, Griffith University, QLD, Australia(信息与通信技术学院,格里菲斯大学,昆士兰,澳大利亚);School of MDP -- Clinical Medicine, Griffith University, QLD, Australia(MDP临床医学学院,格里菲斯大学,昆士兰,澳大利亚)
Domain Generalization for Face Anti-spoofing via Content-aware Composite Prompt Engineering
School of Computer Science, Hefei University of Technology(合肥工业大学计算机学院);State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);School of Electronic Information Engineering, Taiyuan University of Technology(太原理工大学电子信息工程学院);School of Computer Science and Engineering, Faculty of Innovation Engineering, Macau University of Science and Technology(澳门科技大学创新工程学院计算机科学与工程学院);School of Cyber Science and Engineering, Wuhan University(武汉大学网络空间安全学院)
Bayesian Prompt Learning for Image-Language Model Generalization
University of Amsterdam(阿姆斯特丹大学);Samsung AI Cambridge(三星人工智能剑桥研究院);Technical University of Iasi(雅西技术大学);University of Trento(特伦托大学);Queen Mary University of London(伦敦玛丽女王大学)
Transitive Vision-Language Prompt Learning for Domain Generalization
Xiamen University(厦门大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳))
FedBPrompt: Federated Domain Generalization Person Re-Identification via Body Distribution Aware Visual Prompts
School of Computer Science and Technology, Wuhan University of Science and Technology, China(武汉理工大学计算机科学与技术学院,中国);NERC for Multimedia Software, School of Computer Science, Wuhan University, China(武汉大学计算机学院,中国);Nanyang Technological University, Singapore(新加坡南洋理工大学);Changsha Bus Group, China(中国长沙公交集团);Central South University of Forestry and Technology, China(中国林业科技大学);Harbin Institute of Technology Zhengzhou Research Institute, China(哈尔滨工业大学郑州研究院,中国)
Prompt Disentanglement via Language Guidance and Representation Alignment for Domain Generalization
School of Telecommunications Engineering, the State Key Laboratory of Integrated Services Networks (ISN), Xidian University, Xi’an, China(电信工程学院、集成服务网络国家重点实验室(ISN)、西安电子科技大学);Microsoft Research Asia, Shanghai, China(微软亚洲研究院,上海,中国)
HCVP: Leveraging Hierarchical Contrastive Visual Prompt for Domain Generalization
Mohamed bin Zayed University of Artificial Intelligence(莫扎德·本·扎耶德人工智能大学);University of New South Wales(新南威尔士大学);Carnegie Mellon University(卡内基梅隆大学);University of California, San Diego(加州大学圣地亚哥分校);CSIRO’s Data61(CSIRO数据61);University of Sydney(悉尼大学);Macquarie University(麦考瑞大学)
PromptGuard: Soft Prompt-Guided Unsafe Content Moderation for Text-to-Image Models
Department of Computer Science, University of Maryland(计算机科学系,马里兰大学);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算机科学与数据科学学院);Kahlert School of Computing, The University of Utah(犹他大学Kahlert计算学院);Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校Siebel计算与数据科学学院)
Prompt-driven Latent Domain Generalization for Medical Image Classification
Monash University(莫纳什大学);Airdoc-Monash Research, Monash University(Airdoc-莫纳什研究,莫纳什大学);Inception Institute of AI(Inception人工智能研究院);University of Queensland Diamantina Institute, Dermatology Research Centre, The University of Queensland(昆士兰大学Diamantina研究所,皮肤病学研究中心,昆士兰大学);Victorian Melanoma Service, Alfred Health(维多利亚黑色素瘤服务中心,阿尔弗雷德健康中心)
Prompting Decision Transformer for Few-Shot Policy Generalization
Carnegie Mellon University(卡内基梅隆大学);University of Montreal(蒙特利尔大学);Mila(米拉);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室);Massachusetts Institute of Technology(麻省理工学院);UMass Amherst(马萨诸塞大学阿默斯特分校)