我们提出了一种在下游任务上微调大规模视觉-语言预训练模型的新范式,称为Prompt Regularization(ProReg)。与容易对下游任务数据过拟合的传统微调不同,ProReg利用提示预训练模型得到的预测来正则化微调过程。其动机是:通过提示大模型“a photo of a [CLASS]”,填空答案仅依赖于预训练的百科知识,而独立于通常存在偏差的任务数据分布。具体而言,在微调过程中给定训练样本的预测,我们首先计算提示预测的Kullback-Leibler损失和真实标签的Cross-Entropy损失,然后使用提出的样本级自适应权衡权重将它们结合,该权重自动调整预训练域与下游域之间的迁移。在各种out-of-distribution基准上,我们展示了ProReg相较于传统微调、零样本提示、提示调优及其他state-of-the-art方法的一致强性能。
英文摘要:
We present a new paradigm for fine-tuning large-scale visionlanguage pre-trained models on downstream task, dubbed Prompt Regularization (ProReg). Different from traditional fine-tuning which easily overfits to the downstream task data, ProReg uses the prediction by prompting the pretrained model to regularize the fine-tuning. The motivation is: by prompting the large model "a photo of a [CLASS]", the fil-lin answer is only dependent on the pretraining encyclopedic knowledge while independent of the task data distribution, which is usually biased. Specifically, given a training sample prediction during fine-tuning, we first calculate its KullbackLeibler loss of the prompt prediction and Cross-Entropy loss of the ground-truth label, and then combine them with a proposed sample-wise adaptive trade-off weight, which automatically adjusts the transfer between the pretrained and downstream domains. On various out-of-distribution benchmarks, we show the consistently strong performance of ProReg compared with conventional fine-tuning, zero-shot prompt, prompt tuning, and other state-of-the-art methods.
AmPLe: Supporting Vision-Language Models via Adaptive-Debiased Ensemble Multi-Prompt Learning
National Key Laboratory of Space Integrated Information System, Institute of Software, Chinese Academy of Sciences(中国科学院空间信息集成系统国家重点实验室,软件研究所);University of Chinese Academy of Sciences(中国科学院大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学)
Toward Robust Medical Fairness: Debiased Dual-Modal Alignment via Text-Guided Attribute-Disentangled Prompt Learning for Vision-Language Models
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology(集成空天地海大数据应用技术国家工程实验室);Northwestern Polytechnical University(西北工业大学);Huiying Medical Technology Company Ltd.(慧影医疗技术有限公司);The School of Computer Science(计算机学院);The University of Sydney(悉尼大学);Department of Computer Science and Engineering(计算机科学与工程系);The Chinese University of Hong Kong(香港中文大学)
A Similarity Paradigm Through Textual Regularization Without Forgetting
Shanghai Jiao Tong University(上海交通大学);Hong Kong Baptist University(香港浸会大学);Northeastern University(东北大学);University of Science and Technology of China(中国科学技术大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
Prompt-OT: An Optimal Transport Regularization Paradigm for Knowledge Preservation in Vision-Language Model Adaptation
Morgan Stanley(摩根士丹利);Clemson University(克莱姆森大学);Arizona State University(亚利桑那州立大学);Washington University in St. Louis(圣路易斯华盛顿大学);University of Arizona(亚利桑那大学);University of Notre Dame(圣母大学)
DiscoVL: Unveiling Disentangled C ross-Modal Representation Learning via Orthogonal Adversarial Regularization for V ision-Language Models
Shandong Artificial Intelligence Institute(山东人工智能研究院);Qilu University of Technology (Shandong Academy of Sciences)(齐鲁工业大学(山东省科学院));University of Florida(佛罗里达大学)
Few-Shot Image Quality Assessment via Adaptation of Vision-Language Models
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(中国教育部多媒体可信感知与高效计算重点实验室,厦门大学);Beijing Institute of Technology(北京理工大学);Tencent Youtu Lab, Shanghai, China(腾讯优图实验室,中国)
Happy: A Debiased Learning Framework for Continual Generalized Category Discovery
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Centre for Artificial Intelligence and Robotics, HKISI-CAS(香港中文大学(深圳)人工智能与机器人中心(中国科学院香港创新研究院));School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院);School of Computer Science, University of Nottingham(诺丁汉大学计算机学院);MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统实验室)
MAO: Efficient Model-Agnostic Optimization of Prompt Tuning for Vision-Language Models
School of Mechanical Engineering and Automation, Shanghai University(上海大学机械工程与自动化学院);Australian Artificial Intelligence Institute, University of Technology Sydney(悉尼科技大学澳大利亚人工智能学院)
Department of Computer Science and Engineering, Southern University of Science and Technology(南方科技大学计算机科学与工程系);Department of Computer Science and Engineering, Hong Kong University of Science and Technology(香港科技大学计算机科学与工程系);School of Information and Control Engineering, China University of Mining and Technology(中国矿业大学信息与控制工程学院)
Self-regulating Prompts: Foundational Model Adaptation without Forgetting
Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);University of California, Merced(加州大学默塞德分校);Google Research(谷歌研究院);Linköping University(林雪平大学)
Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchors
School of Computer Science and Engineering, University of New South Wales(新南威尔士大学计算机科学与工程学院);Data61, CSIRO(CSIRO数据61研究中心);School of Computer Science, University of Auckland(奥克兰大学计算机科学学院);Australian Institute for Machine Learning (AIML), The University of Adelaide(阿德莱德大学澳大利亚机器学习研究所)
PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discovery
Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Hong Kong University of Science and Technology(香港科技大学);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学);Carnegie Mellon University(卡内基梅隆大学)
Cross Prompting Consistency with Segment Anything Model for Semi-supervised Medical Image Segmentation
The Chinese University of Hong Kong(香港中文大学);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Institute of Medical Intelligence and XR, The Chinese University of Hong Kong(香港中文大学医学智能与XR研究所)
提出 CPC-SAM,在双分支框架中通过 SAM 交叉提示和提示一致性正则化利用标注与未标注数据,显著提升半监督医学图像分割性能。
Bayesian Prompt Learning for Image-Language Model Generalization
University of Amsterdam(阿姆斯特丹大学);Samsung AI Cambridge(三星人工智能剑桥研究院);Technical University of Iasi(雅西技术大学);University of Trento(特伦托大学);Queen Mary University of London(伦敦玛丽女王大学)
Visual Prompt Based Personalized Federated Learning
National University of Defense Technology(国防科技大学);The University of Sydney(悉尼大学);JD Explore Academy(京东探索研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))