arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.16092cs.CVcs.AIcs.LG
收藏

大规模视觉语言模型的并行上下文学习

Parallel In-context Learning for Large Vision Language Models

NTT(日本电信电话研究所)

Shin'ya Yamaguchi, Daiki Chijiwa, Tamao Sakao, Taku Hasegawa

AI总结:

本文提出并行上下文学习方法,通过将长上下文分割为短片段并行处理,提升大视觉语言模型的推理效率,同时保持性能与传统多模态上下文学习相当。

摘要:

大规模视觉语言模型(LVLMs)通过多模态上下文学习(MM-ICL)适应新任务,利用演示示例进行调整。尽管增加演示示例能提升性能,但因Transformer注意力的二次计算成本,导致显著的推理延迟。为解决这一权衡,我们提出了并行上下文学习(Parallel-ICL),一种即插即用的推理算法。Parallel-ICL将长演示上下文分割为多个较短的可管理片段,并并行处理这些片段,在logit层面整合其预测,使用加权产品-of-experts(PoE)集成来近似完整上下文输出。基于集成学习理论,我们引入了并行ICL的原理性策略:(i)基于聚类的上下文分块以最大化片段间多样性,(ii)基于相似性的上下文编译以按查询相关性加权预测。在VQA、图像描述和分类基准上的广泛实验表明,Parallel-ICL在性能上与完整上下文MM-ICL相当,同时显著提高了推理速度。我们的工作为MM-ICL的准确率-效率权衡提供了有效解决方案,使动态任务适应成为可能,同时大幅减少推理开销。

英文摘要:

Large vision-language models (LVLMs) employ multi-modal in-context learning (MM-ICL) to adapt to new tasks by leveraging demonstration examples. While increasing the number of demonstrations boosts performance, they incur significant inference latency due to the quadratic computational cost of Transformer attention with respect to the context length. To address this trade-off, we propose Parallel In-Context Learning (Parallel-ICL), a plug-and-play inference algorithm. Parallel-ICL partitions the long demonstration context into multiple shorter, manageable chunks. It processes these chunks in parallel and integrates their predictions at the logit level, using a weighted Product-of-Experts (PoE) ensemble to approximate the full-context output. Guided by ensemble learning theory, we introduce principled strategies for Parallel-ICL: (i) clustering-based context chunking to maximize inter-chunk diversity and (ii) similarity-based context compilation to weight predictions by query relevance. Extensive experiments on VQA, image captioning, and classification benchmarks demonstrate that Parallel-ICL achieves performance comparable to full-context MM-ICL, while significantly improving inference speed. Our work offers an effective solution to the accuracy-efficiency trade-off in MM-ICL, enabling dynamic task adaptation with substantially reduced inference overhead.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑