PhysToolBench：多模态大语言模型物理工具理解基准
PhysToolBench: Benchmarking Physical Tool Understanding for MLLMs
AI总结：
PhysToolBench是首个评估多模态大语言模型物理工具理解能力的基准，包含超1000个图像-文本对，分三个难度级别，评测32个模型发现显著不足，并提供分析和初步方案。
摘要：
使用、理解和创造工具是人类智能的标志，使人类能够与物理世界进行复杂交互。任何通用智能体要真正实现多功能性，也必须掌握这些基本技能。尽管现代多模态大语言模型（MLLMs）利用其广泛常识在具身AI和下游视觉-语言-动作（VLA）模型中进行高层规划，但它们对物理工具的真实理解程度尚未被量化。为填补这一空白，我们提出PhysToolBench，这是首个专门评估MLLMs物理工具理解能力的基准。该基准构建为视觉问答（VQA）数据集，包含超过1000个图像-文本对。它评估三个不同难度级别的能力：（1）工具识别：要求识别工具的主要功能。（2）工具理解：测试理解工具操作基本原理的能力。（3）工具创造：挑战模型在常规工具不可用时利用周围物体制作新工具。我们对32个MLLMs的全面评估——涵盖专有模型、开源模型、专用具身模型及VLA骨干模型——揭示了工具理解方面的显著不足。此外，我们提供了深入分析并提出了初步解决方案。代码和数据集已公开。
英文摘要：
The ability to use, understand, and create tools is a hallmark of human intelligence, enabling sophisticated interaction with the physical world. For any general-purpose intelligent agent to achieve true versatility, it must also master these fundamental skills. While modern Multimodal Large Language Models (MLLMs) leverage their extensive common knowledge for high-level planning in embodied AI and in downstream Vision-Language-Action (VLA) models, the extent of their true understanding of physical tools remains unquantified. To bridge this gap, we present PhysToolBench, the first benchmark dedicated to evaluating the comprehension of physical tools by MLLMs. Our benchmark is structured as a Visual Question Answering (VQA) dataset comprising over 1,000 image-text pairs. It assesses capabilities across three distinct difficulty levels: (1) Tool Recognition: Requiring the recognition of a tool's primary function. (2) Tool Understanding: Testing the ability to grasp the underlying principles of a tool's operation. (3) Tool Creation: Challenging the model to fashion a new tool from surrounding objects when conventional options are unavailable. Our comprehensive evaluation of 32 MLLMs-spanning proprietary, open-source, specialized embodied, and backbones in VLAs-reveals a significant deficiency in tool understanding. Furthermore, we provide an in-depth analysis and propose preliminary solutions. Code and dataset are publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
