arXiv 2505.248372025-05-30cs.CV

基于层次多粒度图文对齐的零样本中文字符识别

Zero-Shot Chinese Character Recognition with Hierarchical Multi-Granularity Image-Text Aligning

Shanghai Key Laboratory of Intelligent Information Processing（上海智能信息处理关键实验室）；School of Computer Science, Fudan University（复旦大学计算机学院）

Yinglian Zhu, Haiyang Yu, Qizao Wang, Wei Lu, Xiangyang Xue, Bin Li

备注：: The first three authors contributed equally

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对中文字符识别依赖单层表示和后处理的问题，提出基于对比范式的层次多粒度图文对齐框架Hi-GITA，通过多粒度编码器与融合模块提取并融合笔画、部首特征，并利用细粒度解耦图文对比损失对齐图文多粒度表示，在零样本设置下显著提升了识别准确率。

摘要：

中文字符识别（CCR）是智能文档处理的一项基础技术。与拉丁字符不同，汉字展现出独特的空间结构和构字规则，允许在表示中使用细粒度语义信息。然而，现有方法通常基于自回归以及编辑距离后处理，并且典型地依赖单层字符表示。在本文中，我们提出了一种基于对比范式的层次多粒度图文对齐（Hi-GITA）框架。为了利用汉字丰富的细粒度语义信息，我们在图像和文本两侧提出了多粒度编码器。具体而言，图像多粒度编码器从字符图像中提取层次化图像表示，捕获从局部笔画到整体结构的语义线索。文本多粒度编码器在不同粒度级别上提取笔画和部首序列表示。为了更好地捕获笔画和部首之间的关系，我们分别在图像和文本两侧引入了多粒度融合模块。此外，为了有效桥接两种模态，我们进一步引入了细粒度解耦图文对比损失，该损失在多个粒度上对齐图像和文本表示。大量实验表明，我们提出的Hi-GITA显著优于现有的零样本CCR方法。例如，它在手写字符和部首零样本设置中带来了约20%的准确率提升。代码和模型将很快发布。

英文摘要：

Chinese Character Recognition (CCR) is a fundamental technology for intelligent document processing. Unlike Latin characters, Chinese characters exhibit unique spatial structures and compositional rules, allowing for the use of fine-grained semantic information in representation. However, existing approaches are usually based on auto-regressive as well as edit distance post-process and typically rely on a single-level character representation. In this paper, we propose a Hierarchical Multi-Granularity Image-Text Aligning (Hi-GITA) framework based on a contrastive paradigm. To leverage the abundant fine-grained semantic information of Chinese characters, we propose multi-granularity encoders on both image and text sides. Specifically, the Image Multi-Granularity Encoder extracts hierarchical image representations from character images, capturing semantic cues from localized strokes to holistic structures. The Text Multi-Granularity Encoder extracts stroke and radical sequence representations at different levels of granularity. To better capture the relationships between strokes and radicals, we introduce Multi-Granularity Fusion Modules on the image and text sides, respectively. Furthermore, to effectively bridge the two modalities, we further introduce a Fine-Grained Decoupled Image-Text Contrastive loss, which aligns image and text representations across multiple granularities. Extensive experiments demonstrate that our proposed Hi-GITA significantly outperforms existing zero-shot CCR methods. For instance, it brings about 20% accuracy improvement in handwritten character and radical zero-shot settings. Code and models will be released soon.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。