GranViT：面向多模态大语言模型的具有自回归感知的细粒度视觉模型
GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs
AI总结：
本文提出GranViT，一种通过区域级自回归训练和自蒸馏机制实现细粒度感知的视觉Transformer，利用Gran-29M数据集预训练，在细粒度识别、多模态VQA和OCR理解上达到最先进性能。
摘要：
视觉编码器对于多模态大语言模型（MLLMs）在视觉语言任务（如视觉问答和推理）中取得出色性能不可或缺。然而，现有的视觉编码器侧重于全局图像表示，而忽略了细粒度的区域分析。由于细粒度标注数据的稀缺以及缺乏细粒度的预训练范式，它们在细粒度感知方面受到限制。在本文中，我们提出了GranViT，一种新颖的视觉Transformer，通过区域级自回归训练将细粒度特征提取与对大型语言模型（LLMs）的语义对齐相结合。我们首先构建了Gran-29M数据集，该数据集包含200万张自然图像和OCR图像，并配对了超过1.8亿条高质量的区域级标注，以实现大规模细粒度预训练。随后，我们开发了一个预训练-适应框架以及一种自蒸馏机制，以在Gran-29M上训练细粒度的GranViT。我们充分利用Gran-29M中的细粒度标注，采用边界框到标题回归来增强视觉编码器在预训练中的局部视觉表示，以及标题到边界框回归来提高LLM在适应阶段的视觉特征利用和定位能力。我们进一步引入了一种自蒸馏机制，对视觉编码器施加显式的定位约束，以增强其区域推理能力。大量实验表明，GranViT超越了现有的视觉编码器，并展现出对不同LLMs的强大迁移性。值得注意的是，它在细粒度识别、多模态VQA和OCR理解方面取得了最先进的结果。
英文摘要：
Vision encoders are indispensable for allowing impressive performance of Multi-modal Large Language Models (MLLMs) in vision language tasks such as visual question answering and reasoning. However, existing vision encoders focus on global image representations but overlook fine-grained regional analysis. They are limited in fine grained perception due to the scarcity of fine grained annotated data and the lack of a fine grained pre-training paradigm. In this paper, we propose GranViT, a novel Vision Transformer that integrates fine-grained feature extraction with semantic alignment to Large Language Models (LLMs) via region level autoregressive training. We first construct Gran-29M, a dataset comprising 2million natural and OCR images paired with over 180 million high-quality region-level annotations, to enable large scale fine grained pretraining. Consequently, we develop a pretraining-adaptation framework along with a self distillation mechanism to train fine-grained GranViT on Gran-29M. We sufficiently exploit the fine-grained annotations from Gran-29M to resort to bounding-box-to-caption regression to enhance localized visual representation of the vision encoder in the pretraining and caption-to-bounding-box regression to improve vision feature utilization and localization for LLM in the adaptation. We further incorporate a self distillation mechanism that imposes explicit localization constraints on the vision encoder to strengthen its regional reasoning capability. Extensive experiments show that GranViT surpasses existing vision encoders and attains strong transferability to varying LLMs. Remarkably, it achieves state-of-the-art results on fine-grained recognition, multimodal VQA, and OCR understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
