ConES: Concept Embedding Search for Parameter Efficient Tuning Large Vision Language Models
Sichuan University(四川大学);West China Hospital, Sichuan University(四川大学华西医院);Suzhou Institute for Advanced Research, USTC(中国科学技术大学苏州高等研究院);Shanghai AI Laboratory(上海人工智能实验室);Sichuan University Pittsburgh Institute(四川大学匹兹堡学院)
Huahui Yi, Ziyuan Qin, Wei Xu, Miaotian Guo, Kun Wang, Shaoting Zhang, Kang Li, Qicheng Lao
Large pre-trained vision-language models have shown great prominence in transferring pre-acquired knowledge to various domains and downstream tasks with appropriate prompting or tuning. Existing prevalent tuning methods can be generally categorized into three genres: 1) prompt engineering by creating suitable prompt texts, which is time-consuming and requires domain expertise; 2) or simply fine-tuning the whole model, which is extremely inefficient; 3) prompt tuning through parameterized prompt embeddings with the text encoder. Nevertheless, all methods rely on the text encoder for bridging the modality gap between vision and language. In this work, we question the necessity of the cumbersome text encoder for a more lightweight and efficient tuning paradigm as well as more representative prompt embeddings closer to the image representations. To achieve this, we propose a Concept Embedding Search (ConES) approach by optimizing prompt embeddings -- without the need of the text encoder -- to capture the 'concept' of the image modality through a variety of task objectives. By dropping the text encoder, we are able to significantly speed up the learning process, \eg, from about an hour to just ten minutes in our experiments for personalized text-to-image generation without impairing the generation quality. Moreover, our proposed approach is orthogonal to current existing tuning methods since the searched concept embeddings can be further utilized in the next stage of fine-tuning the pre-trained large models for boosting performance. Extensive experiments show that our approach can beat the prompt tuning and textual inversion methods in a variety of downstream tasks including objection detection, instance segmentation, and image generation. Our approach also shows better generalization capability for unseen concepts in specialized domains, such as the medical domain.
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Dalian University of Technology(大连理工大学);Tsinghua University(清华大学)
针对视觉分词器与下游任务优化脱节导致的表示瓶颈,提出 ETT 端到端联合调优方法,在多模态理解与生成任务上较冻结基线提升 2-6%。
MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models
UCAS-Terminus AI Lab, University of Chinese Academic of Sciences(中国科学院大学Terminus AI实验室,中国科学院大学);AIM Lab, University of Amsterdam(阿姆斯特丹大学AIM实验室);Jarvis Research Center, Tencent Youtu Lab(腾讯优图实验室 Jarvis 研究中心);Central Research Institue, United Imaging Healthcare Co., Ltd(联合影像医疗科技有限公司中央研究所)
Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning
School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);Shanghai Engineering Research Center of Intelligent Vision and Imaging(上海智能视觉与成像工程技术研究中心)
FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models
University of Trento(特伦托大学);University of Bergamo(贝拉姆奥大学);Indian Institute of Technology Bombay(印度班加罗尔理工学院);LNMIIT Jaipur(斋普尔LNMIIT);The University of Queensland(昆士兰大学);Fondazione Bruno Kessler(布鲁诺·凯塞勒基金会)
Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning
Southern University of Science and Technology(南方科技大学);The Hong Kong University of Science and Technology(香港科技大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Peng Cheng Laboratory(鹏城实验室)
Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
LLM-empowered Dynamic Prompt Routing for Vision-Language Models Tuning under Long-Tailed Distributions
Shandong University(山东大学);The Hong Kong University of Science and Technology(香港科技大学);AiLF Instruments(AiLF仪器);Shandong Key Laboratory of Intelligent Electronic Packaging Testing and Application(山东省智能电子封装测试与应用重点实验室)
Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Models
Huawei Noah's Ark Lab.(华为诺亚实验室);City University of Hong Kong.(香港城市大学);University of Sydney.(悉尼大学);State Key Lab of General AI, School of Intelligence Science and Technology, Peking University(北京大学人工智能国家重点实验室,智能科学与技术学院)
Towards Fine-Grained Robustness: Attention-Guided Test-Time Prompt Tuning for Vision-Language Models
School of Computer Science and Engineering(计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications(新一代人工智能技术及其交叉应用重点实验室);Southeast University(东南大学);School of Intelligence Science and Engineering(智能科学与工程学院)
Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Following
Department of Electrical and Computer Engineering, The University of British Columbia(英属哥伦比亚大学电气与计算机工程系);Department of Robotics and Mechatronics Engineering, Daegu Gyeongbuk Institute of Science and Technology (DGIST)(大邱庆北科学技术院机器人与机电工程系);Division of Intelligent Robot, Daegu Gyeongbuk Institute of Science and Technology (DGIST)(大邱庆北科学技术院智能机器人系);Vector Institute(向量研究所);Department of Computer Science and Engineering, Pohang University of Science and Technology (POSTECH)(釜山科学技术大学计算机科学与工程系)
From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning
Singapore Eye Research Institute, Singapore National Eye Centre(新加坡眼科研究所、新加坡国家眼科中心);Duke-National University of Singapore Medical School(杜克-新加坡国立大学医学院);Byers Eye Institute, Stanford University(斯坦福大学拜尔斯眼科研究所)
Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models
Southern University of Science and Technology(南方科技大学);The University of Hong Kong(香港大学);Shenzhen Polytechnic University(深圳职业技术大学);ZTE Corporation(中兴通讯股份有限公司);Peng Cheng Laboratory(鹏城实验室)
GLAD: Generalizable Tuning for Vision-Language Models
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院);Northeastern University(东北大学);The Hong Kong Polytechnic University(香港理工大学);Shenzhen University of Advanced Technology(深圳先进技术大学)
Prompt Tuning with Soft Context Sharing for Vision-Language Models
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室);Engineering Laboratory for Intelligent Industrial Vision, CASIA(中国科学院自动化研究所智能工业视觉工程实验室);Research Center of Aerospace Information, CASIA(中国科学院自动化研究所航天信息研究中心)
Remodeling Semantic Relationships in Vision-Language Fine-Tuning
Hangzhou International Innovation Institute, Beihang University(北京航空航天大学杭州国际创新研究院);School of Artificial Intelligence, Beihang University(北京航空航天大学人工智能学院);Nanyang Technological University(南洋理工大学);University of Leicester(莱斯特大学);School of Astronautics, Beihang University(北京航空航天大学航天学院)
MAO: Efficient Model-Agnostic Optimization of Prompt Tuning for Vision-Language Models
School of Mechanical Engineering and Automation, Shanghai University(上海大学机械工程与自动化学院);Australian Artificial Intelligence Institute, University of Technology Sydney(悉尼科技大学澳大利亚人工智能学院)
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);National University of Singapore(新加坡国立大学)