TextHawk：探索多模态大语言模型的高效细粒度感知
TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Models
AI总结：
针对现有多模态大语言模型难适配文档类细粒度感知任务的问题，本文提出TextHawk模型，通过四个专用组件及新构建的指令微调数据集，在文档与通用基准上均超越现有最优方法。
摘要：
多模态大语言模型（Multimodal Large Language Models, MLLMs）已在各类多模态任务上展现出令人瞩目的成果。然而，现有大多数MLLM并不适配面向文档的任务，这类任务需要细粒度图像感知与信息压缩能力。本文提出TextHawk——一款专为面向文档任务设计、同时保留MLLM通用能力的多模态大语言模型。TextHawk旨在通过设计四个专用组件探索高效的细粒度感知：首先，提出重采样与重排列（ReSampling and ReArrangement, ReSA）模块，用于减少文档文本中的冗余、降低MLLM的计算成本；其次，提出可扩展位置嵌入（Scalable Positional Embeddings, SPEs）来编码每个局部特征的位置，可维持不同图像尺寸的可扩展性；随后，采用查询提议网络（Query Proposal Network, QPN）在不同子图像间动态初始化查询；为进一步增强MLLM的细粒度视觉感知能力，设计了多级交叉注意力（Multi-Level Cross-Attention, MLCA）机制，用于捕捉文档图像的层级结构与语义关系。此外，我们借助Gemini Pro丰富多模态文档数据，构建了一个面向文档任务的新型指令微调数据集。我们在通用与面向文档的MLLM基准上开展了大量实验，结果表明TextHawk性能优于现有最优方法，验证了其在细粒度文档感知与通用能力上的有效性与优越性。
英文摘要：
Multimodal Large Language Models (MLLMs) have shown impressive results on various multimodal tasks. However, most existing MLLMs are not well suited for document-oriented tasks, which require fine-grained image perception and information compression. In this paper, we present TextHawk, a MLLM that is specifically designed for document-oriented tasks, while preserving the general capabilities of MLLMs. TextHawk is aimed to explore efficient fine-grained perception by designing four dedicated components. Firstly, a ReSampling and ReArrangement (ReSA) module is proposed to reduce the redundancy in the document texts and lower the computational cost of the MLLM. We explore encoding the positions of each local feature by presenting Scalable Positional Embeddings (SPEs), which can preserve the scalability of various image sizes. A Query Proposal Network (QPN) is then adopted to initialize the queries dynamically among different sub-images. To further enhance the fine-grained visual perceptual ability of the MLLM, we design a Multi-Level Cross-Attention (MLCA) mechanism that captures the hierarchical structure and semantic relations of document images. Furthermore, we create a new instruction-tuning dataset for document-oriented tasks by enriching the multimodal document data with Gemini Pro. We conduct extensive experiments on both general and document-oriented MLLM benchmarks, and show that TextHawk outperforms the state-of-the-art methods, demonstrating its effectiveness and superiority in fine-grained document perception and general abilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
