TextHawk2：一种在双语OCR和定位任务中表现卓越且令牌数减少16倍的大型视觉语言模型
TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokens
AI总结：
针对现有大型视觉语言模型难以同时兼顾OCR与定位性能、细粒度感知令牌消耗高的问题，提出TextHawk2双语模型，通过令牌压缩、视觉编码器强化和数据多样性优化，以16倍更少的图像令牌实现多任务前沿性能，超越同规模闭源模型。
摘要：
读取密集文本并定位图像中的目标是执行高级任务的大型视觉语言模型（LVLM）的基础能力。以往的LVLM，包括GPT-4o这类顶尖专有模型，都难以同时在两项任务中表现出色。此外，以往具备细粒度感知能力的LVLM每张图像需要消耗数千个令牌，导致资源消耗极高。我们提出TextHawk2，这是一款具备高效细粒度感知能力的双语LVLM，在通用任务、OCR任务和定位任务中均展现出前沿性能，且图像令牌数减少了16倍。核心改进包括：（1）令牌压缩：TextHawk2在其前身的高效架构基础上，将每张图像的令牌数大幅减少16倍，以极低资源推动TextHawk系列的训练与部署。（2）视觉编码器强化：我们通过LVLM协同训练增强视觉编码器，释放其在中文OCR和定位等此前未覆盖任务中的潜力。（3）数据多样性：我们保持1亿样本的可比规模，同时实现预训练数据来源的多样化。我们在多个基准上对TextHawk2进行评估，它始终展现出优越性能，超越同规模闭源模型，例如在OCRBench上达到78.4%的准确率，在ChartQA上达到81.4%的准确率，在DocVQA上达到89.6%的ANLS，在RefCOCOg-test上达到88.1%的accuracy@0.5。
英文摘要：
Reading dense text and locating objects within images are fundamental abilities for Large Vision-Language Models (LVLMs) tasked with advanced jobs. Previous LVLMs, including superior proprietary models like GPT-4o, have struggled to excel in both tasks simultaneously. Moreover, previous LVLMs with fine-grained perception cost thousands of tokens per image, making them resource-intensive. We present TextHawk2, a bilingual LVLM featuring efficient fine-grained perception and demonstrating cutting-edge performance across general-purpose, OCR, and grounding tasks with 16 times fewer image tokens. Critical improvements include: (1) Token Compression: Building on the efficient architecture of its predecessor, TextHawk2 significantly reduces the number of tokens per image by 16 times, facilitating training and deployment of the TextHawk series with minimal resources. (2) Visual Encoder Reinforcement: We enhance the visual encoder through LVLM co-training, unlocking its potential for previously unseen tasks like Chinese OCR and grounding. (3) Data Diversity: We maintain a comparable scale of 100 million samples while diversifying the sources of pre-training data. We assess TextHawk2 across multiple benchmarks, where it consistently delivers superior performance and outperforms closed-source models of similar scale, such as achieving 78.4% accuracy on OCRBench, 81.4% accuracy on ChartQA, 89.6% ANLS on DocVQA, and 88.1% accuracy@0.5 on RefCOCOg-test.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
