MapTab: MLLMs 是否已准备好在异构图中进行多标准路线规划？
MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs
AI总结：
本文提出MapTab基准测试，用于评估多模态大语言模型在多标准路线规划任务中的综合推理能力，发现当前模型在多模态推理方面存在显著挑战。
摘要：
系统评估多模态大语言模型（MLLMs）对于推进人工通用智能（AGI）至关重要。然而，现有基准测试仍不足以严格评估其在多标准约束下的推理能力。为弥合这一差距，我们引入MapTab，一个专门设计用于通过路线规划任务评估MLLMs的综合多标准推理能力的多模态基准测试。MapTab要求MLLMs感知并结合地图图像中的视觉线索与结构化表格数据中的路线属性（如时间、价格）。该基准测试涵盖两个场景：Metromap，涵盖52个国家160座城市的地铁网络；Travelmap，描绘19个国家的168个代表性旅游景点。总共包含328张图像、196,800个路线规划查询和3,936个问答查询，所有数据均包含4个关键标准：时间、价格、舒适度和可靠性。对15个代表性MLLMs的广泛评估表明，当前模型在多标准多模态推理方面面临重大挑战。值得注意的是，在视觉感知有限的条件下，多模态协作往往不如单模态方法表现优异。我们认为MapTab提供了一个具有挑战性和现实性的测试平台，以推进MLLMs的系统评估。我们的代码可在https://github.com/Ziqiao-Shang/MapTab上获得。
英文摘要：
Systematically evaluating Multimodal Large Language Models (MLLMs) is essential for advancing Artificial General Intelligence (AGI). Yet existing benchmarks remain inadequate for rigorously measuring their reasoning capabilities under multi-criteria constraints. To address this gap, we introduce MapTab, a multimodal benchmark designed to assess holistic multi-criteria reasoning in MLLMs through route-planning tasks. MapTab requires models to perceive and ground visual information from map images while integrating route attributes, such as Time and Price, from structured tables. It covers two scenarios: Metromap, spanning metro networks in 160 cities across 52 countries, and Travelmap, featuring 168 representative tourist attractions from 19 countries. Overall, MapTab includes 328 images, 196,800 route-planning queries, and 3,936 QA queries, incorporating four key criteria: Time, Price, Comfort, and Reliability. Extensive evaluations of 21 representative MLLMs show that current models still struggle with multicriteria multimodal reasoning. Notably, when visual perception is unreliable, multimodal reasoning can even underperform unimodal approaches. MapTab therefore offers a challenging and realistic testbed for systematically evaluating and advancing MLLMs across core perception, integration, numerical comparison, and route planning capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
