arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.26041cs.CVcs.CL
收藏

OCR推理有多鲁棒?评估视觉语言模型在视觉扰动下的OCR推理鲁棒性

How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations

School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(教育部知识驱动人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际合作中心)

Yuxing Cheng, Yuan Wu, Yi Chang

AI总结:

提出OCR-Robust基准,通过5种扰动类型和3种严重程度评估18个VLM在OCR推理任务上的鲁棒性,发现高干净精度不保证强鲁棒性,图表比文档更脆弱。

摘要:

视觉语言模型(VLM)在基于OCR的基准测试上取得了强劲性能,并日益关注文本丰富的理解,但它们在受控视觉退化下的鲁棒性仍未被充分理解。这一差距对于OCR推理至关重要,因为视觉损坏可能导致OCR错误和结构扭曲,从而给推理任务带来不确定性。为了系统研究这一问题,我们引入了OCR-Robust,一个旨在评估视觉扰动下OCR推理鲁棒性的基准。它包含812个样本,分为两个互补子集:OCR1.0涵盖文档、场景文本、收据、手写和数学内容,OCR2.0专注于图表、几何图形和表格。为了实现高效且信息丰富的评估,我们对18个候选扰动进行了初步研究,并根据它们的影响和跨模型区分性,选择了5种代表性类型,每种类型设置3个严重级别。我们使用干净精度、相对损坏保持率(RCR)、最坏情况保持率(WCR)和复合损坏鲁棒性指数(CRI)来评估鲁棒性,并对18个模型进行基准测试,包括专有系统、开源VLM和OCR+LLM流水线。我们的结果表明,更高的干净精度并不一定意味着更强的鲁棒性,模型在结构敏感的OCR任务上可能在最坏情况下遭受显著退化,并且图表和表格在扰动下比文档类输入脆弱得多。

英文摘要:

Vision-language models (VLMs) have achieved strong performance on OCR-based benchmarks and increasingly focused on text-rich understanding, but their robustness under controlled visual degradation remains insufficiently understood. This gap is critical for OCR reasoning, where visual corruption can induce OCR errors and structural distortions, thereby introducing uncertainty into the reasoning task. To systematically study this problem, we introduce OCR-Robust, a benchmark designed for evaluating OCR reasoning robustness under visual perturbations. It contains 812 samples across two complementary subsets: OCR1.0, covering documents, scene text, receipts, handwriting, and mathematical content, and OCR2.0, focusing on charts, geometry diagrams, and tables. To enable efficient yet informative evaluation, we conduct a pilot study over 18 candidate perturbations and select 5 representative types at 3 severity levels each based on their impact and cross-model discriminability. We evaluate robustness using clean accuracy, Relative Corruption Retention (RCR), Worst-Case Retention (WCR), and a composite Corruption Robustness Index (CRI), and benchmark 18 models spanning proprietary systems, open-source VLMs, and OCR+LLM pipelines. Our results show that higher clean accuracy does not necessarily imply stronger robustness, and that models can suffer pronounced degradation in the worst case on OCR tasks that are sensitive to structure, and charts and tables are substantially more fragile than document-like inputs under perturbation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑