arXiv 2507.022002025-07-02cs.CVcs.AIcs.CL

ESTR-CoT：面向可解释且准确的事件流场景文本识别的思维链推理方法

ESTR-CoT: Towards Explainable and Accurate Event Stream based Scene Text Recognition with Chain-of-Thought Reasoning

School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；School of Electronic and Information Engineering, Anhui University（安徽大学电子与信息学院）；Beijing Institute of Technology（北京理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）

Xiao Wang, Jingtao Jiang, Qiang Chen, Lan Chen, Lin Zhu, Yaowei Wang, Yonghong Tian, Jin Tang

备注：: A Strong Baseline for Reasoning based Event Stream Scene Text Recognition

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出ESTR-CoT框架，将事件流视觉编码与Vicuna-7B大模型结合，通过思维链推理同时输出识别答案和推理过程，并构建大规模CoT数据集，在三个基准上验证了其有效性和可解释性。

摘要：

基于事件流的场景文本识别是近年来新兴的研究课题，在极具挑战性的场景中（尤其是低光照、快速运动）表现优于广泛使用的RGB相机。现有工作要么采用端到端的编码器-解码器框架，要么采用大语言模型来增强识别能力，但仍受限于可解释性不足和上下文逻辑推理能力薄弱等挑战。在本工作中，我们提出了一种新颖的基于思维链推理的事件流场景文本识别框架，称为ESTR-CoT。具体来说，我们首先采用视觉编码器EVA-CLIP（ViT-G/14）将输入事件流转换为token，并利用Llama分词器对给定的生成提示进行编码。使用Q-former将视觉token对齐到预训练大语言模型Vicuna-7B，并同时输出答案和思维链（CoT）推理过程。我们的框架可以通过监督微调以端到端方式进行优化。此外，我们还提出了一个大规模CoT数据集，通过三阶段处理（即生成、润色和专家验证）来训练我们的框架。该数据集为后续基于推理的大模型发展提供了坚实的数据基础。在三个事件流STR基准数据集（即EventSTR、WordArt*、IC15*）上的大量实验充分验证了我们所提出框架的有效性和可解释性。源代码和预训练模型将在https://github.com/Event-AHU/ESTR-CoT上发布。

英文摘要：

Event stream based scene text recognition is a newly arising research topic in recent years which performs better than the widely used RGB cameras in extremely challenging scenarios, especially the low illumination, fast motion. Existing works either adopt end-to-end encoder-decoder framework or large language models for enhanced recognition, however, they are still limited by the challenges of insufficient interpretability and weak contextual logical reasoning. In this work, we propose a novel chain-of-thought reasoning based event stream scene text recognition framework, termed ESTR-CoT. Specifically, we first adopt the vision encoder EVA-CLIP (ViT-G/14) to transform the input event stream into tokens and utilize a Llama tokenizer to encode the given generation prompt. A Q-former is used to align the vision token to the pre-trained large language model Vicuna-7B and output both the answer and chain-of-thought (CoT) reasoning process simultaneously. Our framework can be optimized using supervised fine-tuning in an end-to-end manner. In addition, we also propose a large-scale CoT dataset to train our framework via a three stage processing (i.e., generation, polish, and expert verification). This dataset provides a solid data foundation for the development of subsequent reasoning-based large models. Extensive experiments on three event stream STR benchmark datasets (i.e., EventSTR, WordArt*, IC15*) fully validated the effectiveness and interpretability of our proposed framework. The source code and pre-trained models will be released on https://github.com/Event-AHU/ESTR-CoT.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。