CluMo：用于视觉问答中持续学习的基于聚类的模态融合提示
CluMo: Cluster-based Modality Fusion Prompt for Continual Learning in Visual Question Answering
AI总结：
提出基于聚类的模态融合提示CluMo，通过Key-Key-Prompt对和两阶段训练策略，在视觉问答的多模态持续学习中提升泛化能力并防止灾难性遗忘，达到SOTA性能。
摘要：
大型视觉语言模型（VLMs）在各个应用领域展现出显著的性能提升。然而，采用它们来处理多个顺序遇到的任务一直具有挑战性，因为在某个任务上微调VLM通常会降低其泛化能力和学习新任务的能力，并导致对先前学习任务的灾难性遗忘。在多模态持续学习（CL）设置中启用VLMs有助于应对此类场景。为了提高泛化能力并防止灾难性遗忘，我们提出了一种针对VLMs的基于提示的CL新方法，即基于聚类的模态融合提示（CluMo）。我们设计了一种新颖的Key-Key-Prompt对，其中每个提示都与一个视觉提示键和一个文本提示键相关联。我们采用两阶段训练策略。在第一阶段，通过K-means聚类算法训练单模态键，以帮助选择语义最匹配的提示。在第二阶段，提示键被冻结，选定的提示被附加到输入中，用于在CL场景中训练VLM。在两个基准上的实验表明，我们的方法实现了SOTA性能。
英文摘要：
Large vision-language models (VLMs) have shown significant performance boost in various application domains. However, adopting them to deal with several sequentially encountered tasks has been challenging because finetuning a VLM on a task normally leads to reducing its generalization power and the capacity of learning new tasks as well as causing catastrophic forgetting on previously learned tasks. Enabling using VLMs in multimodal continual learning (CL) settings can help to address such scenarios. To improve generalization capacity and prevent catastrophic forgetting, we propose a novel prompt-based CL method for VLMs, namely $\textbf{Clu}$ster-based $\textbf{Mo}$dality Fusion Prompt (\textbf{CluMo}). We design a novel \textbf{Key-Key-Prompt} pair, where each prompt is associated with a visual prompt key and a textual prompt key. We adopt a two-stage training strategy. During the first stage, the single-modal keys are trained via $K$-means clustering algorithm to help select the best semantically matched prompt. During the second stage, the prompt keys are frozen, the selected prompt is attached to the input for training the VLM in the CL scenario. Experiments on two benchmarks demonstrate that our method achieves SOTA performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
