CLII：通过跨模态预测交互进行视觉-文本修复
CLII: Visual-Text Inpainting via Cross-Modal Predictive Interaction
AI总结：
提出视觉-文本修复任务及跨模态预测交互模型CLII，通过图像与文本分支利用互补信息实现场景文本图像与文本的联合修复，并验证了其在提升SOTA文本检测鲁棒性及超越基线方法上的有效性。
摘要：
图像修复旨在填充受损图像中缺失的像素，并随着前沿学习技术取得了显著进展。然而，最先进的修复方法主要针对自然图像设计，无法正确恢复场景文本图像中的文本，在场景文本图像上训练现有模型也无法解决这些问题。在本研究中，我们提出了视觉-文本修复任务，以实现高质量的场景文本图像恢复和文本补全：给定一张具有未知缺失区域的场景文本图像以及具有未知缺失字符的对应文本，我们的目标是通过利用它们的互补信息来补全图像和文本中缺失的信息。直观地说，输入文本即使受损，也包含图像内容的语言先验，并能指导图像修复。同时，场景文本图像包含字符的外观线索，有利于文本恢复。为此，我们设计了包含两个分支的跨模态预测交互（CLII）模型，即分别用于场景文本修复和文本补全的 ImgBranch 和 TxtBranch，同时有效利用它们的互补性。此外，我们提出将模型嵌入到 SOTA 场景文本检测方法中，显著增强了其对缺失像素的鲁棒性，这证明了新开发任务的实用性。为了验证方法的有效性，我们基于现有的文本相关数据集构建了三个真实数据集，包含 1838 张图像，涵盖弯曲、偶然和风格化文本三种场景，并进行了大量实验，表明我们的方法显著优于基线。
英文摘要：
Image inpainting aims to fill missing pixels in damaged images and has achieved significant progress with cut-edging learning techniques. Nevertheless, state-of-the-art inpainting methods are mainly designed for nature images and cannot correctly recover text within scene text images, and training existing models on the scene text images cannot fix the issues. In this work, we identify the visual-text inpainting task to achieve high-quality scene text image restoration and text completion: Given a scene text image with unknown missing regions and the corresponding text with unknown missing characters, we aim to complete the missing information in both images and text by leveraging their complementary information. Intuitively, the input text, even if damaged, contains language priors of the contents within the images and can guide the image inpainting. Meanwhile, the scene text image includes the appearance cues of the characters that could benefit text recovery. To this end, we design the cross-modal predictive interaction (CLII) model containing two branches, i.e., ImgBranch and TxtBranch, for scene text inpainting and text completion, respectively while leveraging their complementary effectively. Moreover, we propose to embed our model into the SOTA scene text spotting method and significantly enhance its robustness against missing pixels, which demonstrates the practicality of the newly developed task. To validate the effectiveness of our method, we construct three real datasets based on existing text-related datasets, containing 1838 images and covering three scenarios with curved, incidental, and styled texts, and conduct extensive experiments to show that our method outperforms baselines significantly.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
