评估菜单OCR与翻译：一个用于对齐人类与大视觉语言模型自动评估的基准
Evaluating Menu OCR and Translation: A Benchmark for Aligning Human and Automated Evaluations in Large Vision-Language Models
AI总结：
针对现有大视觉语言模型评估忽视复杂布局长文本理解能力的问题，本文提出专门的菜单OCR与翻译基准MOTBench，其自动评估与人类评估高度一致，可用于评估模型性能并指导后续发展。
摘要：
大视觉语言模型（LVLMs）的快速发展极大推动了文档理解领域的应用，尤其是光学字符识别（OCR）和多语言翻译方向。然而，当前对LVLMs的评估（如广泛使用的OCRBench）主要聚焦于验证其短文本响应和简单布局长文本响应的正确性，而对其理解复杂布局长文本能力的评估虽意义重大，却在很大程度上被忽视。本文提出菜单OCR与翻译基准（MOTBench），这是一个专门的评估框架，强调菜单翻译在跨文化交流中的关键作用。MOTBench要求LVLMs准确识别并翻译菜单上的每道菜及其价格和单位项，从而对其视觉理解和语言处理能力进行全面评估。该基准由一批中英文菜单组成，这些菜单布局复杂、字体多样，且包含不同语言的文化特定元素，同时配有精准的人工标注。实验表明，我们的自动评估结果与专业人类评估高度一致。我们评估了一系列公开可用的最先进LVLM，并通过分析其输出明确了其性能的优缺点，为指导LVLM未来的发展提供了有价值的见解。MOTBench可在https://github.com/gitwzl/MOTBench获取。
英文摘要：
The rapid advancement of large vision-language models (LVLMs) has significantly propelled applications in document understanding, particularly in optical character recognition (OCR) and multilingual translation. However, current evaluations of LVLMs, like the widely used OCRBench, mainly focus on verifying the correctness of their short-text responses and long-text responses with simple layout, while the evaluation of their ability to understand long texts with complex layout design is highly significant but largely overlooked. In this paper, we propose Menu OCR and Translation Benchmark (MOTBench), a specialized evaluation framework emphasizing the pivotal role of menu translation in cross-cultural communication. MOTBench requires LVLMs to accurately recognize and translate each dish, along with its price and unit items on a menu, providing a comprehensive assessment of their visual understanding and language processing capabilities. Our benchmark is comprised of a collection of Chinese and English menus, characterized by intricate layouts, a variety of fonts, and culturally specific elements across different languages, along with precise human annotations. Experiments show that our automatic evaluation results are highly consistent with professional human evaluation. We evaluate a range of publicly available state-of-the-art LVLMs, and through analyzing their output to identify the strengths and weaknesses in their performance, offering valuable insights to guide future advancements in LVLM development. MOTBench is available at https://github.com/gitwzl/MOTBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
