arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.11545cs.CVcs.CL
收藏

桥接视觉与语言编码器:面向指代图像分割的参数高效调优

Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation

Sun Yat-sen University(中山大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);AI Institute, Fudan University(复旦大学人工智能研究院);Tencent AI Lab(腾讯人工智能实验室)

Zunnan Xu, Zhihong Chen, Yong Zhang, Yibing Song, Xiang Wan, Guanbin Li

AI总结:

针对指代图像分割的参数高效调优问题,提出新型适配器Bridger与轻量解码器,仅更新少量骨干参数即可在基准上取得相当或更优性能。

摘要:

参数高效调优(Parameter Efficient Tuning, PET)因在保持性能的同时减少参数量、更好地节省硬件资源而受到关注,但目前鲜有研究针对密集预测任务及模态间交互展开。本文针对指代图像分割的高效调优问题开展研究,提出了一种名为Bridger的新型适配器,用于促进跨模态信息交换,并将任务特定信息注入预训练模型中;同时设计了一个用于图像分割的轻量级解码器。在多个具有挑战性的基准上评估表明,该方法仅更新1.61%至3.38%的骨干网络参数,即可取得相当或更优的性能。代码已发布于https://github.com/kkakkkka/ETRIS。

英文摘要:

Parameter Efficient Tuning (PET) has gained attention for reducing the number of parameters while maintaining performance and providing better hardware resource savings, but few studies investigate dense prediction tasks and interaction between modalities. In this paper, we do an investigation of efficient tuning problems on referring image segmentation. We propose a novel adapter called Bridger to facilitate cross-modal information exchange and inject task-specific information into the pre-trained model. We also design a lightweight decoder for image segmentation. Our approach achieves comparable or superior performance with only 1.61\% to 3.38\% backbone parameter updates, evaluated on challenging benchmarks. The code is available at \url{https://github.com/kkakkkka/ETRIS}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑