在多模态模型中利用多尺度对齐推进细粒度视觉理解
Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models
AI总结：
针对多模态大语言模型细粒度知识对齐不足的问题，提出多尺度对齐方法整合文本、坐标与图像，构建超30万条数据合成流水线，并推出约3B参数的TinyGroundingGPT模型，在定位及复杂视觉任务中表现优异。
摘要：
多模态大语言模型（MLLMs）在一系列任务的细粒度视觉理解方面取得了显著成功。然而，由于细粒度知识的对齐不足，它们经常面临重大挑战，这限制了其准确捕捉局部细节和获得全面全局感知的能力。尽管最近的进展集中在将对象表达与定位信息对齐，但它们通常缺乏对对象图像的显式整合，而对象图像包含超出纯文本或坐标的丰富信息。为了弥合这一差距，我们引入了一种新颖的细粒度视觉知识对齐方法，该方法有效地对齐并整合了对象的多尺度知识，包括文本、坐标和图像。这一创新方法以我们的多尺度细粒度增强数据合成流水线为基础，该流水线提供超过30万条必要训练数据，以增强对齐并提升整体性能。此外，我们提出了TinyGroundingGPT，这是一系列针对高级对齐优化的紧凑模型。在约3B参数的规模下，TinyGroundingGPT在定位任务中取得了杰出结果，同时在复杂视觉场景中提供了与更大MLLMs相当的性能。
英文摘要：
Multi-modal large language models (MLLMs) have achieved remarkable success in fine-grained visual understanding across a range of tasks. However, they often encounter significant challenges due to inadequate alignment for fine-grained knowledge, which restricts their ability to accurately capture local details and attain a comprehensive global perception. While recent advancements have focused on aligning object expressions with grounding information, they typically lack explicit integration of object images, which contain affluent information beyond mere texts or coordinates. To bridge this gap, we introduce a novel fine-grained visual knowledge alignment method that effectively aligns and integrates multi-scale knowledge of objects, including texts, coordinates, and images. This innovative method is underpinned by our multi-scale fine-grained enhancement data synthesis pipeline, which provides over 300K essential training data to enhance alignment and improve overall performance. Furthermore, we present TinyGroundingGPT, a series of compact models optimized for high-level alignments. With a scale of approximately 3B parameters, TinyGroundingGPT achieves outstanding results in grounding tasks while delivering performance comparable to larger MLLMs in complex visual scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
