CoVLM：通过通信式解码在大语言模型中组合视觉实体与关系
CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding
AI总结：
针对现有大型视觉语言基础模型组合推理能力不足的问题，本研究提出CoVLM框架，通过通信令牌实现LLM与视觉检测系统的动态迭代通信，在组合推理及传统视觉语言任务上均取得性能提升。
摘要：
人类的一项卓越能力在于组合推理，即“以有限手段实现无限运用”的能力。然而，当前的大型视觉语言基础模型（VLM）缺乏这种组合能力，原因在于它们存在“词袋”行为，且无法构建能正确表征视觉实体及实体间关系的词语。为此，我们提出CoVLM，它可引导大语言模型（LLM）在文本中显式组合视觉实体与关系，并与视觉编码器和检测网络动态通信，实现视觉语言通信式解码。具体而言，我们首先为LLM设计了一组新颖的通信令牌，用于视觉检测系统与语言系统之间的动态通信。LLM在生成一个视觉实体或关系后会生成一个通信令牌，以通知检测网络提出与目前生成的句子相关的区域。随后，这些被提出的感兴趣区域（ROI）被反馈给LLM，以便基于相关区域生成更优的语言。由此，LLM能够通过通信令牌组合视觉实体与关系。视觉到语言与语言到视觉的通信会迭代进行，直到整句生成完毕。我们的框架无缝弥合了视觉感知与LLM之间的差距，在组合推理基准上大幅超越了此前的VLM（例如，HICO-DET的mAP提升约20%，Cola的top-1准确率提升约14%，ARO的top-1准确率提升约3%）。我们还在指代表达理解、视觉问答等传统视觉语言任务上取得了最优性能。
英文摘要：
A remarkable ability of human beings resides in compositional reasoning, i.e., the capacity to make "infinite use of finite means". However, current large vision-language foundation models (VLMs) fall short of such compositional abilities due to their "bag-of-words" behaviors and inability to construct words that correctly represent visual entities and the relations among the entities. To this end, we propose CoVLM, which can guide the LLM to explicitly compose visual entities and relationships among the text and dynamically communicate with the vision encoder and detection network to achieve vision-language communicative decoding. Specifically, we first devise a set of novel communication tokens for the LLM, for dynamic communication between the visual detection system and the language system. A communication token is generated by the LLM following a visual entity or a relation, to inform the detection network to propose regions that are relevant to the sentence generated so far. The proposed regions-of-interests (ROIs) are then fed back into the LLM for better language generation contingent on the relevant regions. The LLM is thus able to compose the visual entities and relationships through the communication tokens. The vision-to-language and language-to-vision communication are iteratively performed until the entire sentence is generated. Our framework seamlessly bridges the gap between visual perception and LLMs and outperforms previous VLMs by a large margin on compositional reasoning benchmarks (e.g., ~20% in HICO-DET mAP, ~14% in Cola top-1 accuracy, and ~3% on ARO top-1 accuracy). We also achieve state-of-the-art performances on traditional vision-language tasks such as referring expression comprehension and visual question answering.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
