面向多模态输入输出的语言模型图像接地
Grounding Language Models to Images for Multimodal Inputs and Outputs
AI总结：
本研究提出一种高效的纯文本预训练语言模型视觉接地方法，通过冻结模型仅微调输入输出线性层实现跨模态交互，可处理交错图文输入并生成带检索图像的文本，在多模态任务上取得优异零样本性能，适用于各类现成语言模型。
摘要：
我们提出一种将预训练纯文本语言模型接地到视觉领域的高效方法，使其能够处理任意交错的图文数据，并生成与检索图像交错的文本。该方法利用语言模型从大规模纯文本预训练中习得的能力，如上下文学习和自由形式文本生成。我们保持语言模型参数冻结，仅微调输入和输出线性层以实现跨模态交互，这使模型能处理任意交错的图文输入，并生成与检索图像交错的自由形式文本。我们在上下文图像检索、多模态对话等接地任务上取得了优异的零样本性能，并展现出出色的交互能力。该方法适用于任何现成的语言模型，为在视觉接地场景中利用预训练语言模型提供了有效通用的解决方案。
英文摘要：
We propose an efficient method to ground pretrained text-only language models to the visual domain, enabling them to process arbitrarily interleaved image-and-text data, and generate text interleaved with retrieved images. Our method leverages the abilities of language models learnt from large scale text-only pretraining, such as in-context learning and free-form text generation. We keep the language model frozen, and finetune input and output linear layers to enable cross-modality interactions. This allows our model to process arbitrarily interleaved image-and-text inputs, and generate free-form text interleaved with retrieved images. We achieve strong zero-shot performance on grounded tasks such as contextual image retrieval and multimodal dialogue, and showcase compelling interactive abilities. Our approach works with any off-the-shelf language model and paves the way towards an effective, general solution for leveraging pretrained language models in visually grounded settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
