超越API：探索多模态大语言模型在物理工具使用中的极限
Beyond APIs: Probing the Limits of MLLMs in Physical Tool Use
AI总结：
提出PhysTool-Bench基准，评估多模态大语言模型在真实场景中识别物理工具并规划使用的能力，发现最强模型仅完成21%任务，揭示感知与规划双重缺陷。
摘要：
多模态大语言模型（MLLMs）在利用数字API方面表现出色，并日益成为具身AI的“大脑”，指导机器人与物理世界交互。在这种具身环境中，核心能力之一是使用物理工具，这支撑着MLLMs在现实任务中协助人类的能力。尽管重要性显著，MLLMs在物理工具使用方面的熟练程度仍 largely unexplored。为填补这一空白，我们引入了PhysTool-Bench，这是首个评估MLLMs理解真实场景、识别物理工具并规划其使用能力的物理工具使用基准。PhysTool-Bench包含2,510个查询，覆盖2,678个真实世界物理工具，涉及制造、电气工程、农业和医疗等多个领域。具体而言，模型沿两个主要维度进行评估：1）识别场景中所有存在的物理工具，2）根据指令和视觉上下文规划工具选择和使用顺序。在13个领先的MLLMs中，即使最强的模型（Gemini-3.1-Pro）也只能识别场景中58.7%的工具，并仅完成21.0%的端到端查询。我们的分析揭示了两个层面的缺陷：MLLMs难以在真实场景中感知工具，而规划阶段更大的下降进一步表明缺乏将感知到的工具映射到任务语义的功能常识，这指出了发展实用具身AI的关键瓶颈。
英文摘要：
Multimodal Large Language Models (MLLMs) excel at utilizing digital APIs and increasingly serve as the "brain" of embodied AI, instructing robots to interact with the physical world. In such embodied settings, a central capability is the use of physical tools, which underpins MLLMs' ability to assist humans in real-world tasks. Despite the importance, MLLMs' proficiency in physical tool use remains largely unexplored. To address this gap, we introduce PhysTool-Bench, the first physical tool-use benchmark designed to evaluate MLLMs' ability to comprehend real-world scenarios, identify physical tools, and plan their use. PhysTool-Bench comprises 2,510 queries over 2,678 real-world physical tools spanning diverse domains, including manufacturing, electrical work, agriculture, and healthcare. Concretely, models are evaluated along two primary dimensions: 1) recognizing all physical tools present in the scene, and 2) planning the tool selection and use sequence based on the instruction and visual context. Across 13 leading MLLMs, even the strongest model (Gemini-3.1-Pro) identifies only 58.7% of tools in a scene and completes merely 21.0% of queries end-to-end. Our analysis reveals a two-level deficit: MLLMs struggle to perceive tools in realistic scenes, and the much larger drop at the planning stage further indicates a lack of functional commonsense for mapping perceived tools onto task semantics, pinpointing a critical bottleneck for the development of practical embodied AI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
