FILA：细粒度视觉语言模型
FILA: Fine-Grained Vision Language Models
AI总结：
针对高分辨率图像裁剪导致的语义断裂问题，提出HyViLM模型，通过设计Hybrid Encoder和最优特征融合策略保留全局上下文，在十项任务中九项优于现有MLLMs。
摘要：
近期，多模态大语言模型（MLLMs）处理高分辨率图像的能力引起了越来越多的关注。目前一种常见的方法是将原始高分辨率图像动态裁剪成较小的子图像，然后将其输入到在较低分辨率图像上预训练的视觉编码器中。然而，这种裁剪方法经常会截断原始图像中的对象和连通区域，导致语义断裂。为了解决这一局限性，我们引入了HyViLM，旨在处理任意分辨率的图像，同时在编码过程中保留整体上下文。具体而言，我们：（i）设计了一种名为Hybrid Encoder的新型视觉编码器，它不仅对单个子图像进行编码，还与详细的全局视觉特征进行交互，显著提升了模型对高分辨率图像的编码能力。（ii）为动态裁剪方法提出了一种最优特征融合策略，有效利用了视觉编码器不同层的信息。在相同设置下与最先进的MLLMs相比，我们的HyViLM在十项任务中的九项上优于现有的MLLMs。具体来说，HyViLM在TextVQA任务上实现了9.6%的性能提升，在DocVQA任务上实现了6.9%的提升。
英文摘要：
Recently, there has been growing interest in the capability of multimodal large language models (MLLMs) to process high-resolution images. A common approach currently involves dynamically cropping the original high-resolution image into smaller sub-images, which are then fed into a vision encoder that was pre-trained on lower-resolution images. However, this cropping approach often truncates objects and connected areas in the original image, causing semantic breaks. To address this limitation, we introduce HyViLM, designed to process images of any resolution while retaining the overall context during encoding. Specifically, we: (i) Design a new visual encoder called Hybrid Encoder that not only encodes individual sub-images but also interacts with detailed global visual features, significantly improving the model's ability to encode high-resolution images. (ii) Propose an optimal feature fusion strategy for the dynamic cropping approach, effectively leveraging information from different layers of the vision encoder. Compared with the state-of-the-art MLLMs under the same setting, our HyViLM outperforms existing MLLMs in nine out of ten tasks. Specifically, HyViLM achieves a 9.6% improvement in performance on the TextVQA task and a 6.9% enhancement on the DocVQA task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
