arXiv 2405.119852025-06-11cs.CV

MTVQA：以文本为中心的多语言视觉问答基准评测

MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering

ByteDance Inc.（字节跳动公司）；Huazhong University of Science and Technology（华中科技大学）；University of Minnesota（明尼苏达大学）；Cornell University（康奈尔大学）

Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, Yangfan He, Kuan Lu, Yanjie… 展开作者

Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, Yangfan He, Kuan Lu, Yanjie Wang, Yuliang Liu, Hao Liu, Xiang Bai, Can Huang

备注：: Accepted by ACL 2025 findings

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出覆盖9种语言、含2,116张图像和6,778个专家标注问答对的MTVQA基准，评测多种MLLM并提供训练数据，揭示多语言文本中心视觉理解仍有显著提升空间。

摘要：

格式规范的以文本为中心的视觉问答（Text-Centric Visual Question Answering，TEC-VQA）不仅有助于在以文本为中心的视觉环境中实现人机交互，还可作为事实上的黄金代理，用于评估AI模型在以文本为中心的场景理解领域的能力。尽管如此，现有大多数TEC-VQA基准都集中于英语和中文等高资源语言。尽管已有开创性工作通过翻译引擎在非以文本为中心的VQA数据集中扩展多语言问答对，但这种基于翻译的方案在应用于TEC-VQA时会遇到严重的“视觉-文本错位”问题。具体而言，它优先考虑问答对中的文本，而忽视图像中呈现的视觉文本。此外，该方案也无法处理与细微语义、语境失真、语言偏倚以及问题类型多样性相关的复杂情况。在本工作中，我们通过推出MTVQA来应对多语言TEC-VQA问题；MTVQA是首个在9种不同语言上具备高质量人工专家标注的基准，包含覆盖2,116张图像的6,778个问答对。此外，通过在MTVQA基准上全面评估众多先进的多模态大语言模型（MLLM），包括Qwen2-VL、GPT-4o、GPT-4V、Claude3和Gemini，可以明显看出其性能仍有很大提升空间（Qwen2-VL得分为30.9，而人类表现为79.7），这凸显了MTVQA的价值。另外，我们在MTVQA数据集中提供了多语言训练数据，并证明使用这些数据进行简单微调即可显著提升多语言TEC-VQA性能。我们希望MTVQA将为研究界提供新的见解，并推动对多语言视觉文本理解的进一步探索。项目主页位于 https://bytedance.github.io/MTVQA/。

英文摘要：

Text-Centric Visual Question Answering (TEC-VQA) in its proper format not only facilitates human-machine interaction in text-centric visual environments but also serves as a de facto gold proxy to evaluate AI models in the domain of text-centric scene understanding. Nonetheless, most existing TEC-VQA benchmarks have focused on high-resource languages like English and Chinese. Despite pioneering works to expand multilingual QA pairs in non-text-centric VQA datasets through translation engines, the translation-based protocol encounters a substantial "visual-textual misalignment" problem when applied to TEC-VQA. Specifically, it prioritizes the text in question-answer pairs while disregarding the visual text present in images. Moreover, it fails to address complexities related to nuanced meaning, contextual distortion, language bias, and question-type diversity. In this work, we tackle multilingual TEC-VQA by introducing MTVQA, the first benchmark featuring high-quality human expert annotations across 9 diverse languages, consisting of 6,778 question-answer pairs across 2,116 images. Further, by comprehensively evaluating numerous state-of-the-art Multimodal Large Language Models~(MLLMs), including Qwen2-VL, GPT-4o, GPT-4V, Claude3, and Gemini, on the MTVQA benchmark, it is evident that there is still a large room for performance improvement (Qwen2-VL scoring 30.9 versus 79.7 for human performance), underscoring the value of MTVQA. Additionally, we supply multilingual training data within the MTVQA dataset, demonstrating that straightforward fine-tuning with this data can substantially enhance multilingual TEC-VQA performance. We aspire that MTVQA will offer the research community fresh insights and stimulate further exploration in multilingual visual text comprehension. The project homepage is available at https://bytedance.github.io/MTVQA/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。