面向场景文本识别的关联对比学习与掩码图像建模
Relational Contrastive Learning and Masked Image Modeling for Scene Text Recognition
AI总结：
本文提出RCMSTR框架，通过关联对比学习和掩码图像建模建模文本元素间丰富关联，并引入解耦设计与稀疏卷积，在多个场景文本识别下游任务中超越现有自监督方法。
摘要：
在监督式场景文本识别（STR）中，利用单词的语义先验，上下文感知方法已经取得了显著进展。考虑到STR中文本与背景的异质性，我们提出，这种上下文先验可以被重新解释为文本元素之间的关联，从而作为表示学习的有效自监督标签。然而，由于词汇依赖性，文本关联被限制在数据集的有限规模内，这会导致过拟合问题，从而损害表示质量。为了解决这一问题，我们的工作引入了一个面向STR的关联对比学习与掩码图像建模统一框架（RCMSTR），该框架显式地对丰富的文本关联进行建模。对于RCL分支，我们首先引入关联重排模块，以在训练过程中动态生成新的关联。在此基础上，我们进一步进行关联对比学习，以对帧、子词和词之间的层级内与层级间关联进行建模。另一方面，MIM可以通过掩码自然地增强上下文信息，我们发现块掩码策略对STR更为有效。为了有效整合RCL与MIM，我们还引入了一种新颖的解耦设计，旨在减轻掩码图像对对比学习的影响。此外，为了增强MIM与CNN的兼容性，我们提出采用稀疏卷积，并在训练中直接与密集卷积共享权重。所提出的RCMSTR在不同STR相关下游任务的各种评估协议中表现出优越的性能，优于现有的最先进的自监督STR技术。消融研究和定性实验结果进一步验证了我们方法的有效性。代码和预训练模型将在 https://github.com/ThunderVVV/RCMSTR 上提供。
英文摘要：
Context-aware methods have achieved remarkable advancements in supervised scene text recognition by leveraging semantic priors from words. Considering the heterogeneity of text and background in STR, we propose that such contextual priors can be reinterpreted as the relations between textual elements, serving as effective self-supervised labels for representation learning. However, textual relations are restricted to the finite size of the dataset due to lexical dependencies, which causes over-fitting problem, thus compromising the representation quality. To address this, our work introduces a unified framework of Relational Contrastive Learning and Masked Image Modeling for STR (RCMSTR), which explicitly models the enriched textual relations. For the RCL branch, we first introduce the relational rearrangement module to cultivate new relations on the fly. Based on this, we further conduct relational contrastive learning to model the intra- and inter-hierarchical relations for frames, sub-words and words. On the other hand, MIM can naturally boost the context information via masking, where we find that the block masking strategy is more effective for STR. For the effective integration of RCL and MIM, we also introduce a novel decoupling design aimed at mitigating the impact of masked images on contrastive learning. Additionally, to enhance the compatibility of MIM with CNNs, we propose the adoption of sparse convolutions and directly sharing the weights with dense convolutions in training. The proposed RCMSTR demonstrates superior performance in various evaluation protocols for different STR-related downstream tasks, outperforming the existing state-of-the-art self-supervised STR techniques. Ablation studies and qualitative experimental results further validate the effectiveness of our method. The code and pre-trained models will be available at https://github.com/ThunderVVV/RCMSTR .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
