IAP：通过实例感知提示改进视觉-语言模型的持续学习
IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting
AI总结：
本文针对多域任务增量学习中提示设计优化问题，提出实例感知提示框架，通过实例级门控提示和类分布驱动提示提升新任务适应能力并缓解遗忘。
摘要：
近年来，预训练视觉-语言模型（PT-VLM）在实践中经常面临多域任务增量学习（MTIL）场景，其中多模态任务的若干类别和领域会逐步到达。由于无法访问先前见过的任务和未见过的任务，受内存约束的MTIL会遭受前向遗忘和后向遗忘。为了缓解上述挑战，参数高效微调技术（PEFT），例如提示微调，被用于使PT-VLM适应多样化的增量学习任务。为了实现有效的新任务适应，现有方法仅考虑PEFT策略选择的影响，却忽视了PEFT参数设置（例如提示方式）的影响。在本文中，我们应对MTIL中为多样化任务优化提示设计的挑战，并提出实例感知提示（IAP）框架。具体而言，我们的实例感知门控提示（IA-GP）策略通过在实例级别自适应地跨Transformer层分配提示，增强对新任务的适应能力，同时缓解遗忘。我们的实例感知类分布驱动提示（IA-CDDP）通过为每个实例确定准确的任务标签相关置信度分数，改进任务适应过程。在11个数据集上使用三种性能指标进行的实验评估证明了我们所提方法的有效性。源代码可在https://github.com/FerdinandZJU/IAP获取。
英文摘要：
Recent pre-trained vision-language models (PT-VLMs) often face a Multi-Domain Task Incremental Learning (MTIL) scenario in practice, where several classes and domains of multi-modal tasks are incrementally arrived. Without access to previously seen tasks and unseen tasks, memory-constrained MTIL suffers from forward and backward forgetting. To alleviate the above challenges, parameter-efficient fine-tuning techniques (PEFT), such as prompt tuning, are employed to adapt the PT-VLM to the diverse incrementally learned tasks. To achieve effective new task adaptation, existing methods only consider the effect of PEFT strategy selection, but neglect the influence of PEFT parameter setting (e.g., prompting). In this paper, we tackle the challenge of optimizing prompt designs for diverse tasks in MTIL and propose an Instance-Aware Prompting (IAP) framework. Specifically, our Instance-Aware Gated Prompting (IA-GP) strategy enhances adaptation to new tasks while mitigating forgetting by adaptively assigning prompts across transformer layers at the instance level. Our Instance-Aware Class-Distribution-Driven Prompting (IA-CDDP) improves the task adaptation process by determining an accurate task-label-related confidence score for each instance. Experimental evaluations across 11 datasets, using three performance metrics, demonstrate the effectiveness of our proposed method. The source codes are available at https://github.com/FerdinandZJU/IAP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
