迷失在翻译与噪声中：对VLMs在现实表格上的失败模式的深入探讨
Lost in Translation and Noise: A Deep Dive into the Failure Modes of VLMs on Real-World Tables
AI总结：
MirageTVQA通过多语言和视觉不完美的表格挑战VLMs，揭示了模型在现实噪声和语言转换中的失败模式。
摘要：
VLMs的出色表现主要是在无法捕捉真实世界复杂性的基准测试上衡量的。现有的表格问答数据集，如WikiTableQuestions和FinQA，大多是单语（英语）的，并以数字完美、干净的格式呈现表格。这造成了研究与实践之间的显著差距。为了解决这个问题，我们提出了MirageTVQA，一个新基准，用于评估VLMs在这两个维度上的表现。该基准包含近60,000个问答对，涵盖24种语言，挑战模型的表格不仅多语言，而且视觉上不完美，包含现实噪声以模仿扫描文档。我们对领先VLMs的评估揭示了两个主要失败点：当面对视觉噪声时，性能严重下降（最佳模型下降超过35%），以及一种持续的英语优先偏差，其中推理能力无法转移到其他语言。MirageTVQA为衡量和推动更稳健的VLM模型在表格推理中的进步提供了基准。数据集和代码可在：https://github.com/anshulsc/MirageTVQA获取。
英文摘要：
The impressive performance of VLMs is largely measured on benchmarks that fail to capture the complexities of real-world scenarios. Existing datasets for tabular QA, such as WikiTableQuestions and FinQA, are overwhelmingly monolingual (English) and present tables in a digitally perfect, clean format. This creates a significant gap between research and practice. To address this, we present \textbf{MirageTVQA}, a new benchmark designed to evaluate VLMs on these exact dimensions. Featuring nearly 60,000 QA pairs across 24 languages, MirageTVQA challenges models with tables that are not only multilingual but also visually imperfect, incorporating realistic noise to mimic scanned documents. Our evaluation of the leading VLMs reveals two primary failure points: a severe degradation in performance (over 35\% drop for the best models) when faced with visual noise and a consistent English-first bias where reasoning abilities fail to transfer to other languages. MirageTVQA provides a benchmark for measuring and driving progress towards more robust VLM models for table reasoning. The dataset and the code are available at: https://github.com/anshulsc/MirageTVQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
