低秩提示交互用于持续视觉-语言检索
Low-rank Prompt Interaction for Continual Vision-Language Retrieval
AI总结：
提出低秩提示交互（LPI）方法，同时建模跨模态与跨任务交互，通过低秩分解和对比学习提升持续视觉-语言检索性能，仅需极少量参数。
摘要：
多模态任务中的持续学习研究正受到越来越多的关注。然而，大多数现有工作忽视了显式的跨模态和跨任务交互。在本文中，我们创新性地提出了低秩提示交互（LPI）来解决这一多模态理解中的普遍问题，该方法同时考虑了跨模态和跨任务交互。具体而言，对于前者，我们在相应的Transformer层中采用多模态关联模块。考虑到训练参数随层数和任务数扩展，我们提出低秩交互增强分解，以避免内存爆炸，同时通过共享和分离公共-特定低秩因子来增强跨模态关联。此外，由于低秩初始化携带的多模态语义差异，我们采用分层低秩对比学习以确保训练鲁棒性。对于后者，我们首先进行视觉分析，发现不同任务在邻近性上存在明显差异。因此，我们在提示学习过程中基于任务语义距离引入显式的任务对比约束。在两个检索任务上的实验表明，引入极少量参数即可实现性能提升，证明了我们方法的有效性。代码可在 https://github.com/Kelvin-ywc/LPI 获取。
英文摘要：
Research on continual learning in multi-modal tasks has been receiving increasing attention. However, most existing work overlooks the explicit cross-modal and cross-task interactions. In this paper, we innovatively propose the Low-rank Prompt Interaction (LPI) to address this general problem of multi-modal understanding, which considers both cross-modal and cross-task interactions. Specifically, as for the former, we employ multi-modal correlation modules for corresponding Transformer layers. Considering that the training parameters scale to the number of layers and tasks, we propose low-rank interaction-augmented decomposition to avoid memory explosion while enhancing the cross-modal association through sharing and separating common-specific low-rank factors. In addition, due to the multi-modal semantic differences carried by the low-rank initialization, we adopt hierarchical low-rank contrastive learning to ensure training robustness. As for the latter, we initially employ a visual analysis and identify that different tasks have clear distinctions in proximity. Therefore, we introduce explicit task contrastive constraints in the prompt learning process based on task semantic distances. Experiments on two retrieval tasks show performance improvements with the introduction of a minimal number of parameters, demonstrating the effectiveness of our method. Code is available at https://github.com/Kelvin-ywc/LPI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
