LLaVA-SG：在视觉语言模型中利用场景图作为视觉语义表达
LLaVA-SG: Leveraging Scene Graphs as Visual Semantic Expression in Vision-Language Models
AI总结：
本文提出LLaVA-SG，通过在视觉语言模型中引入场景图表达模块，解决ViT图像分块导致的碎片化感知问题，提升模型的视觉语义理解能力。
摘要：
近期大型视觉语言模型（VLMs）的进展通常采用基于Vision Transformer（ViT）架构的视觉编码器。ViT将图像划分为图像块的方式导致了碎片化的感知，从而阻碍了VLMs的视觉理解能力。在本文中，我们提出了一种创新性增强方法，通过在VLMs中引入场景图表达（Scene Graph Expression，SGE）模块来解决这一局限性。该模块提取并结构化地表达图像中复杂的语义信息，从而提升VLMs的基础感知与理解能力。大量实验表明，集成我们的SGE模块能显著增强VLM在视觉语言任务中的性能，表明其在保留复杂语义细节和促进更好的视觉理解方面的有效性。
英文摘要：
Recent advances in large vision-language models (VLMs) typically employ vision encoders based on the Vision Transformer (ViT) architecture. The division of the images into patches by ViT results in a fragmented perception, thereby hindering the visual understanding capabilities of VLMs. In this paper, we propose an innovative enhancement to address this limitation by introducing a Scene Graph Expression (SGE) module in VLMs. This module extracts and structurally expresses the complex semantic information within images, thereby improving the foundational perception and understanding abilities of VLMs. Extensive experiments demonstrate that integrating our SGE module significantly enhances the VLM's performance in vision-language tasks, indicating its effectiveness in preserving intricate semantic details and facilitating better visual understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
