mPLUG-DocOwl2：面向无OCR多页文档理解的高分辨率压缩
mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding
AI总结：
本文提出DocOwl2，通过High-resolution DocCompressor将高分辨率文档图像压缩至324个token，并经三阶段训练实现高效无OCR多页文档理解，显著降低延迟且保持单页性能。
摘要：
多模态大语言模型（MLLMs）通过提高文档图像支持的分辨率，在无OCR文档理解方面取得了可观性能。然而，其代价是单张文档图像会生成数千个视觉token，导致GPU显存占用过高、推理速度变慢，在多页文档理解中尤为突出。为应对这些挑战，本文提出High-resolution DocCompressor模块，在低分辨率全局视觉特征引导下，将每张高分辨率文档图像压缩为324个token。基于该压缩模块，为增强多页文档理解能力并兼顾token效率与问答性能，作者在三阶段训练框架下开发了DocOwl2：单图像预训练、多图像持续预训练和多任务微调。DocOwl2在多个多页文档理解基准上创下新的最优水平，并将首token延迟降低50%以上，展现出先进的多页问答、带证据页面解释以及跨页结构理解能力。此外，与在相似数据上训练的单图像MLLM相比，DocOwl2以不足20%的视觉token取得了相当的单页理解性能。代码、模型和数据已公开发布于https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2。
英文摘要：
Multimodel Large Language Models(MLLMs) have achieved promising OCR-free Document Understanding performance by increasing the supported resolution of document images. However, this comes at the cost of generating thousands of visual tokens for a single document image, leading to excessive GPU memory and slower inference times, particularly in multi-page document comprehension. In this work, to address these challenges, we propose a High-resolution DocCompressor module to compress each high-resolution document image into 324 tokens, guided by low-resolution global visual features. With this compression module, to strengthen multi-page document comprehension ability and balance both token efficiency and question-answering performance, we develop the DocOwl2 under a three-stage training framework: Single-image Pretraining, Multi-image Continue-pretraining, and Multi-task Finetuning. DocOwl2 sets a new state-of-the-art across multi-page document understanding benchmarks and reduces first token latency by more than 50%, demonstrating advanced capabilities in multi-page questioning answering, explanation with evidence pages, and cross-page structure understanding. Additionally, compared to single-image MLLMs trained on similar data, our DocOwl2 achieves comparable single-page understanding performance with less than 20% of the visual tokens. Our codes, models, and data are publicly available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
