arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.09489cs.CL
收藏

面向视觉推理任务的 Q-Former 高效视觉-语言对齐

Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Tasks

Weavel, Inc.(韦弗公司);Seoul National University(首尔大学);UC Berkeley(加州大学伯克利分校);Graduate School of Data Science, Seoul National University(首尔大学数据科学研究生院)

Sungkyung Kim, Adam Lee, Junyoung Park, Andrew Chung, Jusang Oh, Jay-Yoon Lee

AI总结:

本研究探讨了使用 InstructBLIP 对 Q-Former 进行参数高效微调(PEFT),发现仅用不到 2% 的参数即可达到全量微调效果,并通过 AdaLoRA 揭示了自注意力层在视觉推理任务中更为重要。

摘要:

大型语言模型的最新进展表明,通过采用额外的编码器来对齐不同模态,其在视觉推理任务中的能力得到了增强。虽然 Q-Former 已被广泛用作通用编码器,用于将包括图像、视频、音频和 3D 在内的多种模态与大型语言模型对齐,但先前关于其高效训练及对其各个组件分析的研究仍然有限。在本研究中,我们使用 InstructBLIP 结合视觉推理基准 ScienceQA 和 IconQA,研究了参数高效微调(PEFT)Q-Former 的有效性。我们观察到,对 Q-Former 应用 PEFT 可实现与全量微调相当的性能,且仅使用不到 2% 的可训练参数。此外,我们采用 AdaLoRA 进行动态参数预算重新分配,以在 4 个不同的基准上检验 Q-Former 各子层的相对重要性。我们的发现表明,在感知性视觉-语言推理任务中,自注意力层明显更为重要,而 FFN 层的相对重要性则取决于任务中涉及的视觉-语言模式的复杂性。代码可在 https://github.com/AttentionX/InstructBLIP_PEFT 获取。

英文摘要:

Recent advancements in large language models have demonstrated enhanced capabilities in visual reasoning tasks by employing additional encoders for aligning different modalities. While the Q-Former has been widely used as a general encoder for aligning several modalities including image, video, audio, and 3D with large language models, previous works on its efficient training and the analysis of its individual components have been limited. In this work, we investigate the effectiveness of parameter efficient fine-tuning (PEFT) the Q-Former using InstructBLIP with visual reasoning benchmarks ScienceQA and IconQA. We observe that applying PEFT to the Q-Former achieves comparable performance to full fine-tuning using under 2% of the trainable parameters. Additionally, we employ AdaLoRA for dynamic parameter budget reallocation to examine the relative importance of the Q-Former's sublayers with 4 different benchmarks. Our findings reveal that the self-attention layers are noticeably more important in perceptual visual-language reasoning tasks, and relative importance of FFN layers depends on the complexity of visual-language patterns involved in tasks. The code is available at https://github.com/AttentionX/InstructBLIP_PEFT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑