arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.09712cs.CVcs.AI
收藏

LAST:利用工具作为提示以增强多模态大语言模型的空间推理能力

LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models

National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室)

Shi-Yu Tian, Zhi Zhou, Kun-Yang Yu, Ming Yang, Yang Chen, Ziqiao Shang, Lan-Zhe Guo, Yu-Feng Li

AI总结:

LAST通过整合专用视觉模型,解决多模态大语言模型在复杂几何布局解析中的幻觉和不精确问题,提出统一框架提升空间推理能力。

摘要:

空间推理是智能系统感知和与物理世界交互的核心能力。然而,多模态大语言模型(MLLMs)在解析复杂几何布局时经常出现幻觉和不精确。由于数据驱动的扩展难以内部化结构几何先验和空间约束,整合成熟的专用视觉模型成为有吸引力的替代方案。尽管有潜力,将此范式应用于空间推理受到两个关键挑战的阻碍:调用异构、参数丰富的工具的难度,以及理解并有效利用其多样化的低级输出(例如分割掩码、深度图)在高层推理中的挑战。为了解决这些挑战,我们提出了LAST,一个统一的工具增强空间推理框架。LAST包含一个可扩展的交互沙盒,称为LAST-Box,它将异构工具调用抽象为原子指令和可重用的空间技能,返回多模态提示(例如标注图像和文本描述),这些提示可以直接被LLMs消费。我们进一步设计了三阶段渐进训练策略,引导模型从理解工具输出到熟练和适应的工具调用。在四个数据集上的实验表明,LAST-7B在基准上比其基础模型提升了约20%,并优于强大的专有闭源LLMs,显著增强了复杂空间任务的推理能力。

英文摘要:

Spatial reasoning is a cornerstone capability for intelligent systems to perceive and interact with the physical world. However, multimodal large language models (MLLMs) frequently suffer from hallucinations and imprecision when parsing complex geometric layouts. As data-driven scaling struggles to internalize structured geometric priors and spatial constraints, integrating mature, specialized vision models presents a compelling alternative. Despite its promise, applying this paradigm to spatial reasoning is hindered by two key challenges: The difficulty of invoking heterogeneous, parameter-rich tools, as well as the challenge of understanding and effectively leveraging their diverse low-level outputs (e.g., segmentation masks, depth maps) in high-level reasoning. To address these challenges, we propose LAST, a unified framework for tool-augmented spatial reasoning. LAST features an extensible interactive sandbox, termed LAST-Box, which abstracts heterogeneous tool invocations into atomic instructions and reusable spatial skills, returning multimodal hints (e.g., annotated images and textual descriptions) that can be directly consumed by LLMs. We further design a three-stage progressive training strategy that guides models from understanding tool outputs to proficient and adaptive tool invocation. Experiments on four datasets show that LAST-7B achieves around 20\% performance gains over its backbone and outperforms strong proprietary closed-source LLMs, substantially enhancing reasoning on complex spatial tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑