arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.18993cs.CV
收藏

ConES:面向大视觉语言模型参数高效调优的概念嵌入搜索

ConES: Concept Embedding Search for Parameter Efficient Tuning Large Vision Language Models

Sichuan University(四川大学);West China Hospital, Sichuan University(四川大学华西医院);Suzhou Institute for Advanced Research, USTC(中国科学技术大学苏州高等研究院);Shanghai AI Laboratory(上海人工智能实验室);Sichuan University Pittsburgh Institute(四川大学匹兹堡学院)

Huahui Yi, Ziyuan Qin, Wei Xu, Miaotian Guo, Kun Wang, Shaoting Zhang, Kang Li, Qicheng Lao

AI总结:

本文提出ConES方法,通过无需文本编码器的概念嵌入搜索实现大视觉语言模型的参数高效调优,在目标检测、实例分割和图像生成等任务中超越现有提示调优方法,并显著加速训练过程。

摘要:

大型预训练视觉-语言模型在通过适当的提示或调优将预获取知识迁移到各种领域和下游任务方面表现出显著优势。现有主流调优方法通常可分为三类:1)通过创建合适的提示文本进行提示工程,这耗时且需要领域专业知识;2)或简单地微调整个模型,这极其低效;3)通过文本编码器使用参数化提示嵌入进行提示调优。然而,所有这些方法都依赖文本编码器来弥合视觉与语言之间的模态差距。在这项工作中,我们质疑笨重的文本编码器对于更轻量、更高效的调优范式以及更接近图像表示的更具代表性的提示嵌入的必要性。为了实现这一目标,我们提出了一种概念嵌入搜索(ConES)方法,通过优化提示嵌入——无需文本编码器——来通过多种任务目标捕获图像模态的“概念”。通过去掉文本编码器,我们能够显著加速学习过程,例如,在我们的个性化文本到图像生成实验中,从大约一小时缩短到仅十分钟,且不损害生成质量。此外,我们提出的方法与现有调优方法正交,因为搜索到的概念嵌入可以在下一阶段的预训练大模型微调中进一步利用以提升性能。大量实验表明,我们的方法在包括目标检测、实例分割和图像生成在内的多种下游任务中能够超越提示调优和文本反演方法。我们的方法在专业领域(如医学领域)的未见概念上也表现出更好的泛化能力。

英文摘要:

Large pre-trained vision-language models have shown great prominence in transferring pre-acquired knowledge to various domains and downstream tasks with appropriate prompting or tuning. Existing prevalent tuning methods can be generally categorized into three genres: 1) prompt engineering by creating suitable prompt texts, which is time-consuming and requires domain expertise; 2) or simply fine-tuning the whole model, which is extremely inefficient; 3) prompt tuning through parameterized prompt embeddings with the text encoder. Nevertheless, all methods rely on the text encoder for bridging the modality gap between vision and language. In this work, we question the necessity of the cumbersome text encoder for a more lightweight and efficient tuning paradigm as well as more representative prompt embeddings closer to the image representations. To achieve this, we propose a Concept Embedding Search (ConES) approach by optimizing prompt embeddings -- without the need of the text encoder -- to capture the 'concept' of the image modality through a variety of task objectives. By dropping the text encoder, we are able to significantly speed up the learning process, \eg, from about an hour to just ten minutes in our experiments for personalized text-to-image generation without impairing the generation quality. Moreover, our proposed approach is orthogonal to current existing tuning methods since the searched concept embeddings can be further utilized in the next stage of fine-tuning the pre-trained large models for boosting performance. Extensive experiments show that our approach can beat the prompt tuning and textual inversion methods in a variety of downstream tasks including objection detection, instance segmentation, and image generation. Our approach also shows better generalization capability for unseen concepts in specialized domains, such as the medical domain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑