arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.11248cs.CV
收藏

OCR-VQGAN:驯服图像内文本生成

OCR-VQGAN: Taming Text-within-Image Generation

Computer Vision Center(计算机视觉中心);ServiceNow Research(ServiceNow研究院);ÉTS Montréal(蒙特利尔高等技术学院)

Juan A. Rodriguez, David Vazquez, Issam Laradji, Marco Pedersoli, Pau Rodriguez

摘要:

合成图像生成最近在诸如自然图像或艺术生成的领域中经历了显著的改进。然而,图形和图表生成的问题仍然是未开发的。生成图形和图表的一个挑战性方面是有效地渲染图像中的可读文本。为了缓解这个问题,我们提出了OCR-VQGAN,一种图像编码器和解码器,它利用OCR预训练的特征来优化文本感知损失,鼓励架构保留高保真文本和图表结构。为了探索我们的方法,我们引入了Paper 2Fig 100 k数据集,其中包含来自研究论文的超过100 k的图形和文本图像。这些图显示了可在arXiv.org上获得的来自人工智能和计算机视觉等领域的文章的架构图和方法。图形通常包括文本和离散对象,例如,图中的框,用线和箭头连接它们。我们证明了OCR-VQGAN的有效性进行了几个实验的任务,图形重建。此外,我们探讨了在整体损失函数中加权不同感知度量的定性和定量影响。我们在https://github.com/joanrod/ocr-vqgan上发布代码、模型和数据集。

英文摘要:

Synthetic image generation has recently experienced significant improvements in domains such as natural image or art generation. However, the problem of figure and diagram generation remains unexplored. A challenging aspect of generating figures and diagrams is effectively rendering readable texts within the images. To alleviate this problem, we present OCR-VQGAN, an image encoder, and decoder that leverages OCR pre-trained features to optimize a text perceptual loss, encouraging the architecture to preserve high-fidelity text and diagram structure. To explore our approach, we introduce the Paper2Fig100k dataset, with over 100k images of figures and texts from research papers. The figures show architecture diagrams and methodologies of articles available at arXiv.org from fields like artificial intelligence and computer vision. Figures usually include text and discrete objects, e.g., boxes in a diagram, with lines and arrows that connect them. We demonstrate the effectiveness of OCR-VQGAN by conducting several experiments on the task of figure reconstruction. Additionally, we explore the qualitative and quantitative impact of weighting different perceptual metrics in the overall loss function. We release code, models, and dataset at https://github.com/joanrod/ocr-vqgan.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑