漫步多模态领域：通过低秩序列多模态适配器实现高效迁移学习
A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter
AI总结：
针对现有高效迁移学习方法难适配多模态场景、模态交互利用不足且效率低的问题，提出低秩序列多模态适配器Wander，通过外积融合、CP分解和令牌级低秩分解实现高效多模态交互，在多模态数据集上性能优于现有最优方法。
摘要：
基于适配器的高效迁移学习方法已在单模态模型和视觉-语言模型中取得巨大成功。然而，现有方法在微调多模态模型时存在两大主要挑战：其一，它们是为视觉-语言任务设计的，无法扩展到模态数量超过两种的场景；其二，它们对模态间交互的利用程度有限，且效率不足。为解决这些问题，本文提出了低秩序列多模态适配器Wander。我们首先利用外积以逐元素方式有效融合不同模态的信息；为提升效率，采用CP分解将张量分解为秩一分量，大幅减少了参数量；此外，我们还实现了令牌级低秩分解，以提取更细粒度的特征及模态间的序列关系。凭借这些设计，Wander能够以参数高效的方式实现不同模态序列间的令牌级交互。我们在包含不同模态数量的数据集上开展了大量实验，结果显示Wander的性能始终优于当前最优的高效迁移学习方法，充分证明了其有效性、高效性与通用性。
英文摘要：
Efficient transfer learning methods such as adapter-based methods have shown great success in unimodal models and vision-language models. However, existing methods have two main challenges in fine-tuning multimodal models. Firstly, they are designed for vision-language tasks and fail to extend to situations where there are more than two modalities. Secondly, they exhibit limited exploitation of interactions between modalities and lack efficiency. To address these issues, in this paper, we propose the loW-rank sequence multimodal adapter (Wander). We first use the outer product to fuse the information from different modalities in an element-wise way effectively. For efficiency, we use CP decomposition to factorize tensors into rank-one components and achieve substantial parameter reduction. Furthermore, we implement a token-level low-rank decomposition to extract more fine-grained features and sequence relationships between modalities. With these designs, Wander enables token-level interactions between sequences of different modalities in a parameter-efficient way. We conduct extensive experiments on datasets with different numbers of modalities, where Wander outperforms state-of-the-art efficient transfer learning methods consistently. The results fully demonstrate the effectiveness, efficiency and universality of Wander.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
