arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.01131cs.CV
收藏

M2IST:面向高效指代表达理解的多模态交互式旁路调优

M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension

College of Electronics and Information Engineering, Sichuan University(四川大学电子信息学院);College of Systems Engineering, National University of Defense Technology(国防科技大学系统工程学院);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Engineering, Westlake University(西湖大学工程学院);College of Computer Science and Cyber Security (Pilot Software College), Chengdu University of Technology(成都理工大学计算机与网络安全学院(软件学院));Yunnan Key Laboratory of Software Engineering, Yunnan University(云南大学云南省软件工程重点实验室)

Xuyang Liu, Ting Liu, Siteng Huang, Yi Xin, Yue Hu, Quanjun Yin, Donglin Wang, Yuanyuan Wu, Honggang Chen

AI总结:

本文提出M2IST多模态交互式旁路调优方法,通过固定预训练编码器并仅更新M3ISAs适配器,在仅用2.11%可调参数的情况下实现高效指代表达理解,兼顾性能与效率。

摘要:

指代表达理解(REC)是一项视觉-语言任务,旨在根据语言表达在图像中定位目标物体。对通用预训练视觉-语言基础模型进行全量微调用于REC可获得出色性能,但成本日益高昂。参数高效迁移学习(PETL)方法已展现出以更少可调参数实现强劲性能的能力。然而,直接将PETL应用于REC面临两个挑战:(1)预训练视觉-语言基础模型之间的多模态交互不足;(2)由于梯度流经庞大的视觉-语言基础模型,导致GPU显存占用过高。为此,我们提出M2IST:多模态交互式旁路调优,并引入M3ISAs:多模态交互式旁路适配器混合体。在微调过程中,我们固定预训练的单模态编码器,仅更新M3ISAs,以实现面向REC的高效视觉-语言对齐。实验结果表明,M2IST在性能-效率权衡上优于全量微调和其他PETL方法,仅需2.11%的可调参数、39.61%的GPU显存和63.46%的训练时间,同时保持有竞争力的性能。我们的代码已发布于https://github.com/xuyang-liu16/M2IST。

英文摘要:

Referring expression comprehension (REC) is a vision-language task to locate a target object in an image based on a language expression. Fully fine-tuning general-purpose pre-trained vision-language foundation models for REC yields impressive performance but becomes increasingly costly. Parameter-efficient transfer learning (PETL) methods have shown strong performance with fewer tunable parameters. However, directly applying PETL to REC faces two challenges: (1) insufficient multi-modal interaction between pre-trained vision-language foundation models, and (2) high GPU memory usage due to gradients passing through the heavy vision-language foundation models. To this end, we present M2IST: Multi-Modal Interactive Side-Tuning with M3ISAs: Mixture of Multi-Modal Interactive Side-Adapters. During fine-tuning, we fix the pre-trained uni-modal encoders and update M3ISAs to enable efficient vision-language alignment for REC. Empirical results reveal that M2IST achieves better performance-efficiency trade-off than full fine-tuning and other PETL methods, requiring only 2.11\% tunable parameters, 39.61\% GPU memory, and 63.46\% training time while maintaining competitive performance. Our code is released at https://github.com/xuyang-liu16/M2IST.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑