基于CLIP的对称语言特征蒸馏用于场景文本识别
Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognition
AI总结：
本文提出CLIP-OCR框架，利用对称蒸馏策略和语言一致性损失，从CLIP中提取视觉与语言知识，在六个场景文本识别基准上达到93.8%平均准确率。
摘要：
本文探索了对比语言-图像预训练（CLIP）模型在场景文本识别（STR）中的潜力，并建立了一个新颖的对称语言特征蒸馏框架（名为CLIP-OCR），以利用CLIP中的视觉和语言知识。与以往主要考虑视觉编码特征泛化的基于CLIP的方法不同，我们提出了一种对称蒸馏策略（SDS），进一步捕获CLIP文本编码器中的语言知识。通过将CLIP图像编码器与反转的CLIP文本编码器级联，构建了一个对称结构，具有从图像到文本的特征流，不仅涵盖视觉信息，还涵盖用于蒸馏的语言信息。得益于CLIP中的自然对齐，这种引导流提供了从视觉到语言的渐进优化目标，可以逐层监督STR特征转发过程。此外，提出了一种新的语言一致性损失（LCL），通过在优化过程中考虑二阶统计量来增强语言能力。总体而言，CLIP-OCR是首个为STR任务设计图像与文本之间平滑过渡的方法。大量实验证明了CLIP-OCR的有效性，在六个流行的STR基准上平均准确率达到93.8%。代码将在https://github.com/wzx99/CLIPOCR提供。
英文摘要：
In this paper, we explore the potential of the Contrastive Language-Image Pretraining (CLIP) model in scene text recognition (STR), and establish a novel Symmetrical Linguistic Feature Distillation framework (named CLIP-OCR) to leverage both visual and linguistic knowledge in CLIP. Different from previous CLIP-based methods mainly considering feature generalization on visual encoding, we propose a symmetrical distillation strategy (SDS) that further captures the linguistic knowledge in the CLIP text encoder. By cascading the CLIP image encoder with the reversed CLIP text encoder, a symmetrical structure is built with an image-to-text feature flow that covers not only visual but also linguistic information for distillation.Benefiting from the natural alignment in CLIP, such guidance flow provides a progressive optimization objective from vision to language, which can supervise the STR feature forwarding process layer-by-layer.Besides, a new Linguistic Consistency Loss (LCL) is proposed to enhance the linguistic capability by considering second-order statistics during the optimization. Overall, CLIP-OCR is the first to design a smooth transition between image and text for the STR task.Extensive experiments demonstrate the effectiveness of CLIP-OCR with 93.8% average accuracy on six popular STR benchmarks.Code will be available at https://github.com/wzx99/CLIPOCR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
