AIM:让任意多模态大语言模型拥抱高效上下文学习
AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning
AI总结:
本文提出轻量框架AIM,通过将多模态演示的图像信息聚合为融合虚拟token,使任意MLLM无需微调即可高效进行上下文学习,解决了多图演示读取困难和视觉token开销大的问题。
摘要:
上下文学习(ICL)使大语言模型(LLMs)无需更新数十亿参数即可在下游任务中展现涌现能力。然而,在多模态大语言模型(MLLMs)领域,有两个问题阻碍了多模态上下文学习的应用:(1)大多数主流MLLMs仅在单图像数据集上训练,使其无法读取多模态演示。(2)随着演示数量增加,数千个视觉token对硬件构成严峻挑战,并降低ICL性能。在初步探索中,我们发现内部LLM在多模态演示中更倾向于关注语言模态以生成响应。因此,我们提出一个通用且轻量化的框架AIM,通过将多模态演示的图像信息聚合到相应语言部分的稠密潜在空间中来解决上述问题。具体而言,AIM首先使用冻结的主干MLLM读取每个图像-文本演示,并在文本之上提取向量表示。这些向量自然融合了图像-文本对的信息,AIM通过一个可训练投影层将其转换为内部LLM可接受的融合虚拟token。最终,这些融合token作为多模态演示的变体,像往常一样输入MLLM以引导其对当前查询的响应。由于这些融合token源自图像-文本对的文本成分,多模态演示几乎被简化为纯文本演示,从而无缝适用于任何MLLM。在MLLM实际保持冻结的情况下,AIM具有参数高效性,我们在与下游测试任务无关的公开多模态网络语料上对其进行训练。
英文摘要:
In-context learning (ICL) facilitates Large Language Models (LLMs) exhibiting emergent ability on downstream tasks without updating billions of parameters. However, in the area of multi-modal Large Language Models (MLLMs), two problems hinder the application of multi-modal ICL: (1) Most primary MLLMs are only trained on single-image datasets, making them unable to read multi-modal demonstrations. (2) With the demonstrations increasing, thousands of visual tokens highly challenge hardware and degrade ICL performance. During preliminary explorations, we discovered that the inner LLM tends to focus more on the linguistic modality within multi-modal demonstrations to generate responses. Therefore, we propose a general and light-weighted framework \textbf{AIM} to tackle the mentioned problems through \textbf{A}ggregating \textbf{I}mage information of \textbf{M}ultimodal demonstrations to the dense latent space of the corresponding linguistic part. Specifically, AIM first uses the frozen backbone MLLM to read each image-text demonstration and extracts the vector representations on top of the text. These vectors naturally fuse the information of the image-text pair, and AIM transforms them into fused virtual tokens acceptable for the inner LLM via a trainable projection layer. Ultimately, these fused tokens function as variants of multi-modal demonstrations, fed into the MLLM to direct its response to the current query as usual. Because these fused tokens stem from the textual component of the image-text pair, a multi-modal demonstration is nearly reduced to a pure textual demonstration, thus seamlessly applying to any MLLMs. With its de facto MLLM frozen, AIM is parameter-efficient and we train it on public multi-modal web corpora which have nothing to do with downstream test tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。