通过跨模态适配器将高效且通用的视觉知识整合进预训练语言模型
Towards Versatile and Efficient Visual Knowledge Integration into Pre-trained Language Models with Cross-Modal Adapters
AI总结：
本文提出即插即用模块X-adapter，通过V-expert和T-expert子模块融合VLMs的图像与文本表示并注入PLMs，仅更新新增参数，显著提升了物体颜色推理和NLU任务性能。
摘要：
人类通过多模态知识学习语言。然而，由于纯文本的预训练方案，大多数现有的预训练语言模型（PLMs）无法获取多模态信息。为了将视觉知识注入PLMs，现有方法要么引入视觉语言模型（VLMs）的文本编码器，要么引入图像编码器来编码视觉信息，并更新PLMs的所有原始参数以进行知识融合。在本文中，我们提出了一种新的即插即用模块X-adapter，以灵活利用在预训练VLMs中学习到的对齐的视觉和文本知识，并将其高效地注入PLMs。具体而言，我们将X-adapters插入PLMs中，在适应阶段仅更新新增的参数。为了充分挖掘VLMs的潜力，X-adapters包含两个子模块：V-expert和T-expert，分别用于融合VLMs的图像和文本表示。我们可以根据下游任务选择激活不同的子模块。实验结果表明，与PLM基线相比，我们的方法能显著提升在物体颜色推理和自然语言理解（NLU）任务上的性能。
英文摘要：
Humans learn language via multi-modal knowledge. However, due to the text-only pre-training scheme, most existing pre-trained language models (PLMs) are hindered from the multi-modal information. To inject visual knowledge into PLMs, existing methods incorporate either the text or image encoder of vision-language models (VLMs) to encode the visual information and update all the original parameters of PLMs for knowledge fusion. In this paper, we propose a new plug-and-play module, X-adapter, to flexibly leverage the aligned visual and textual knowledge learned in pre-trained VLMs and efficiently inject them into PLMs. Specifically, we insert X-adapters into PLMs, and only the added parameters are updated during adaptation. To fully exploit the potential in VLMs, X-adapters consist of two sub-modules, V-expert and T-expert, to fuse VLMs' image and text representations, respectively. We can opt for activating different sub-modules depending on the downstream tasks. Experimental results show that our method can significantly improve the performance on object-color reasoning and natural language understanding (NLU) tasks compared with PLM baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
