对抗训练与OCR模态扰动用于场景文本视觉问答
Adversarial Training with OCR Modality Perturbation for Scene-Text Visual Question Answering
AI总结：
针对场景文本视觉问答中OCR错误和空间信息利用不足的问题，提出含对抗性OCR增强和空间感知自注意力的多模态对抗训练架构，在ST-VQA和TextVQA上显著提升性能。
摘要：
场景文本视觉问答（ST-VQA）旨在理解图像中的场景文本并回答与文本内容相关的问题。大多数现有方法严重依赖光学字符识别（OCR）系统的准确性，而基于有限的空间位置信息和错误的OCR文本信息进行激进微调往往会导致不可避免的过拟合。在本文中，我们提出了一种具有空间感知能力的多模态对抗训练架构。具体而言，我们引入了一个对抗性OCR增强（AOE）模块，该模块利用OCR模态嵌入空间中的对抗训练来增强OCR文本的容错表示，从而减少由OCR错误引起的噪声。同时，我们添加了一种空间感知自注意力（SASA）机制，以帮助模型更好地捕捉OCR标记之间的空间关系。多项实验表明，我们的方法在ST-VQA和TextVQA数据集上均取得了显著的性能提升，并为多模态对抗训练提供了一种新颖的范式。
英文摘要：
Scene-Text Visual Question Answering (ST-VQA) aims to understand scene text in images and answer questions related to the text content. Most existing methods heavily rely on the accuracy of Optical Character Recognition (OCR) systems, and aggressive fine-tuning based on limited spatial location information and erroneous OCR text information often leads to inevitable overfitting. In this paper, we propose a multimodal adversarial training architecture with spatial awareness capabilities. Specifically, we introduce an Adversarial OCR Enhancement (AOE) module, which leverages adversarial training in the embedding space of OCR modality to enhance fault-tolerant representation of OCR texts, thereby reducing noise caused by OCR errors. Simultaneously, We add a Spatial-Aware Self-Attention (SASA) mechanism to help the model better capture the spatial relationships among OCR tokens. Various experiments demonstrate that our method achieves significant performance improvements on both the ST-VQA and TextVQA datasets and provides a novel paradigm for multimodal adversarial training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
