arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.09875cs.CV
收藏

面向大视觉语言模型的选择性状态空间记忆

Selective State Space Memory for Large Vision-Language Models

Universiti Teknologi Malaysia(马来西亚工艺大学)

Chee Ng, Yuen Fung

AI总结:

针对大视觉语言模型领域微调计算成本高的问题,提出基于Mamba的轻量状态空间记忆整合方法SSMI,仅更新少量参数即可在多个多模态基准上取得SOTA性能,兼具效率与泛化性。

摘要:

大视觉语言模型(LVLM)已在各类多模态任务中展现出卓越性能,但针对特定领域应用微调这类模型仍是一项计算密集型挑战。本文提出状态空间记忆整合(SSMI),一种用于LVLM高效微调的新方法。通过将轻量级基于Mamba的状态空间模块集成到LVLM架构中,SSMI可捕获长程依赖,并有效注入任务特定的视觉与序列模式。与传统微调方法不同,SSMI仅需更新模型的一小部分参数,具备计算高效性与可扩展性。在COCO Captioning、VQA、Flickr30k等基准数据集上的实验表明,SSMI在保持鲁棒性与泛化能力的同时取得了当前最优性能。综合分析进一步验证了SSMI在效率、适配性与可解释性方面的优势,是大规模视觉语言模型微调的理想方案。

英文摘要:

Large Vision-Language Models (LVLMs) have demonstrated remarkable performance across a wide range of multimodal tasks. However, fine-tuning these models for domain-specific applications remains a computationally intensive challenge. This paper introduces State Space Memory Integration (SSMI), a novel approach for efficient fine-tuning of LVLMs. By integrating lightweight Mamba-based state space modules into the LVLM architecture, SSMI captures long-range dependencies and injects task-specific visual and sequential patterns effectively. Unlike traditional fine-tuning methods, SSMI requires only a fraction of the model's parameters to be updated, making it computationally efficient and scalable. Experiments on benchmark datasets, including COCO Captioning, VQA, and Flickr30k, demonstrate that SSMI achieves state-of-the-art performance while maintaining robustness and generalization capabilities. Comprehensive analysis further validates the advantages of SSMI in terms of efficiency, adaptability, and interpretability, positioning it as a compelling solution for fine-tuning large-scale vision-language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑