指令引导的场景文本识别
Instruction-Guided Scene Text Recognition
AI总结：
本文提出指令引导场景文本识别范式IGTR，通过预测字符属性实现文本图像理解，在保持小模型和快推理的同时显著超越现有方法，并有效解决罕见字符和形近字识别难题。
摘要：
多模态模型在视觉识别任务中展现出引人注目的性能，因为自由形式的文本引导训练激发了理解细粒度视觉内容的能力。然而，由于自然图像与文本图像之间的组成差异，现有模型无法直接应用于场景文本识别（STR）。我们提出了一种新颖的指令引导场景文本识别（IGTR）范式，该范式将STR形式化为指令学习问题，并通过预测字符属性（例如字符频率、位置等）来理解文本图像。IGTR首先设计$\u003c条件，问题，答案\u003e$指令三元组，提供丰富多样的字符属性描述。为了通过问答有效学习这些属性，IGTR开发了一个轻量级指令编码器、一个跨模态特征融合模块和一个多任务答案头，从而引导细粒度的文本图像理解。此外，IGTR仅通过使用不同指令即可实现不同的识别流程，实现了一种基于字符理解的文本推理范式，与当前方法有显著差异。在英文和中文基准上的实验表明，IGTR以显著优势超越现有模型，同时保持较小的模型规模和较快的推理速度。此外，通过调整指令采样，IGTR提供了一种优雅的方式来处理罕见出现和形态相似字符的识别，这些在以往都是挑战。代码：https://github.com/Topdu/OpenOCR。
英文摘要：
Multi-modal models have shown appealing performance in visual recognition tasks, as free-form text-guided training evokes the ability to understand fine-grained visual content. However, current models cannot be trivially applied to scene text recognition (STR) due to the compositional difference between natural and text images. We propose a novel instruction-guided scene text recognition (IGTR) paradigm that formulates STR as an instruction learning problem and understands text images by predicting character attributes, e.g., character frequency, position, etc. IGTR first devises $\left \langle condition,question,answer\right \rangle$ instruction triplets, providing rich and diverse descriptions of character attributes. To effectively learn these attributes through question-answering, IGTR develops a lightweight instruction encoder, a cross-modal feature fusion module and a multi-task answer head, which guides nuanced text image understanding. Furthermore, IGTR realizes different recognition pipelines simply by using different instructions, enabling a character-understanding-based text reasoning paradigm that differs from current methods considerably. Experiments on English and Chinese benchmarks show that IGTR outperforms existing models by significant margins, while maintaining a small model size and fast inference speed. Moreover, by adjusting the sampling of instructions, IGTR offers an elegant way to tackle the recognition of rarely appearing and morphologically similar characters, which were previous challenges. Code: https://github.com/Topdu/OpenOCR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
