超越OCR精度：图像退化条件下以文本为中心的VQA——模块化与端到端方法比较

中文摘要 AI 辅助

以文本为中心的视觉问答（VQA）需要读取并推理图像中嵌入的文本，当图像遭受运动模糊、低分辨率或压缩伪影等真实世界退化时，该任务会变得显著更加困难。尽管模块化的基于OCR的流程和端到端的视觉语言模型都被广泛用于此任务，但它们在退化条件下的相对鲁棒性仍未得到充分探索。我们提出了一项实证研究，比较了两种模块化流程与SA-DBNet（一种结合ResNet-18与自注意力空间建模和可变形卷积的自定义检测器架构）以及一个端到端的视觉语言基线，在4013张退化图像和7000个问答对上进行了评估。微调后的模块化流程实现了高达57.50%的精确匹配准确率，而端到端基线为38.00%，领域特定微调带来了高达29.50个百分点的增益。关键的是，我们发现传统的OCR错误指标（如字符错误率和词错误率）无法可靠预测下游VQA性能，因为当存在上下文线索时，语义推理可以补偿识别失败。这些发现强调了在真实视觉条件下对以文本为中心的VQA系统进行任务感知评估的重要性。代码可在此处获取。

英文摘要

Text-centric Visual Question Answering (VQA) requires reading and reasoning over text embedded in images, a task made substantially harder when images suffer from real-world degradation such as motion blur, low resolution, or compression artifacts. While modular OCR-based pipelines and end-to-end vision-language models are both widely used for this task, their comparative robustness under degraded conditions remains underexplored. We present an empirical study comparing two modular pipelines with SA-DBNet, a custom detector architecture combining ResNet-18 with self-attention spatial modeling and deformable convolutions against an end-to-end vision-language baseline, evaluated on 4013 degraded images with 7000 question-answer pairs. Fine-tuned modular pipelines achieve up to 57.50% exact-match accuracy versus 38.00% for the end-to-end baseline, with domain-specific fine-tuning yielding a gain of up to 29.50 percentage points. Critically, we find that conventional OCR error metrics like Character Error Rate and Word Error Rate are unreliable predictors of downstream VQA performance, as semantic reasoning can compensate for recognition failures when contextual cues are present. These findings highlight the importance of task-aware evaluation for text-centric VQA systems under realistic visual conditions. Codes are available here

发表机构

• Indian Institute of Technology Mandi（印度理工学院曼迪分校）
• DIT University（DIT大学）
• Indian Institute of Technology Kharagpur（印度理工学院卡拉格普尔分校）

机构由 AI 辅助整理，请以论文原文为准。