arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.07082cs.CVcs.AI
收藏

MosaicThinker: 通过迭代构建空间表示实现设备端具身AI的视觉空间推理

MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representation

Department of Electrical and Computer Engineering(电气与计算机工程系);University of Pittsburgh(匹兹堡大学)

Haoming Wang, Qiyao Xue, Weichen Liu, Wei Gao

AI总结:

MosaicThinker通过迭代构建空间表示,提升设备端具身AI在复杂跨帧空间推理任务中的准确性。

摘要:

当具身AI从传统的物体检测和识别扩展到更高级的机器人操作和执行规划任务时,从视频输入中进行视觉空间推理是感知物体空间关系并指导设备动作的必要条件。然而,现有的视觉语言模型(VLMs)在空间推理方面能力非常有限,由于缺乏对3D空间信息的知识,尤其是在涉及跨多帧视频的复杂空间关系时更为明显。在本文中,我们提出了一种新的推理时计算技术,即MosaicThinker,用于设备端的具身AI,旨在增强小型VLM在困难的跨帧推理任务上的空间推理能力。我们的基本思想是将多个帧中的碎片化空间信息整合到统一的全局语义地图空间表示中,并进一步通过视觉提示引导VLM在语义地图上的空间推理。实验结果表明,我们的技术能够显著提高在资源受限的具身AI设备上跨帧空间推理的准确性,适用于类型和复杂度各不相同的推理任务。

英文摘要:

When embodied AI is expanding from traditional object detection and recognition to more advanced tasks of robot manipulation and actuation planning, visual spatial reasoning from the video inputs is necessary to perceive the spatial relationships of objects and guide device actions. However, existing visual language models (VLMs) have very weak capabilities in spatial reasoning due to the lack of knowledge about 3D spatial information, especially when the reasoning task involve complex spatial relations across multiple video frames. In this paper, we present a new inference-time computing technique for on-device embodied AI, namely \emph{MosaicThinker}, which enhances the on-device small VLM's spatial reasoning capabilities on difficult cross-frame reasoning tasks. Our basic idea is to integrate fragmented spatial information from multiple frames into a unified space representation of global semantic map, and further guide the VLM's spatial reasoning over the semantic map via a visual prompt. Experiment results show that our technique can greatly enhance the accuracy of cross-frame spatial reasoning on resource-constrained embodied AI devices, over reasoning tasks with diverse types and complexities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑