PositionOCR: 通过混合专家整合增强多模态模型的位置意识
PositionOCR: Augmenting Positional Awareness in Multi-Modal Models via Hybrid Specialist Integration
AI总结：
PositionOCR通过整合文本定位专家与LLM的上下文能力，提升多模态模型在文本定位和定位任务中的位置准确性。
摘要：
近年来，多模态大语言模型（MLLMs）在以OCR为中心的视觉问答（VQA）任务中取得了强劲的表现，展示了其处理异构数据和在不同上下文中表现出适应性的能力。然而，这些MLLMs依赖于一个大型语言模型（LLM）作为解码器，而LLM主要是为语言处理设计的，因此本质上缺乏处理精确视觉任务所需的位置推理能力，例如文本定位和文本定位。此外，MLLMs的大量参数需要大量的计算资源和大规模数据进行有效训练。相反，文本定位专家实现了最先进的坐标预测，但缺乏语义推理能力。这种二元性促使我们的核心研究问题：能否将专家的效率与LLM的上下文能力相结合，创造一个位置准确的MLLM？为了克服这些挑战，我们引入了PositionOCR，一种参数高效的混合架构，无缝整合文本定位模型的位置优势与LLM的上下文推理能力。该框架包含131M可训练参数，展示了出色的多模态处理能力，特别是在文本定位和文本定位任务中表现突出，持续超越传统MLLMs。
英文摘要：
In recent years, Multi-modal Large Language Models (MLLMs) have achieved strong performance in OCR-centric Visual Question Answering (VQA) tasks, illustrating their capability to process heterogeneous data and exhibit adaptability across varied contexts. However, these MLLMs rely on a Large Language Model (LLM) as the decoder, which is primarily designed for linguistic processing, and thus inherently lacks the positional reasoning required for precise visual tasks, such as text spotting and text grounding. Additionally, the extensive parameters of MLLMs necessitate substantial computational resources and large-scale data for effective training. Conversely, text spotting specialists achieve state-of-the-art coordinate predictions but lack semantic reasoning capabilities. This dichotomy motivates our key research question: Can we synergize the efficiency of specialists with the contextual power of LLMs to create a positionally-accurate MLLM? To overcome these challenges, we introduce PositionOCR, a parameter-efficient hybrid architecture that seamlessly integrates a text spotting model's positional strengths with an LLM's contextual reasoning. Comprising 131M trainable parameters, this framework demonstrates outstanding multi-modal processing capabilities, particularly excelling in tasks such as text grounding and text spotting, consistently surpassing traditional MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
