arXiv 2502.131302025-02-18cs.CVcs.AIcs.HCcs.LGcs.RO

Magma：面向多模态 AI 智能体的基础模型

Magma: A Foundation Model for Multimodal AI Agents

Microsoft Research（微软研究院）；University of Maryland（马里兰大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；KAIST（韩国科学技术院）；University of Washington（华盛顿大学）

Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, Yuquan Deng, Lars Liden, Jianfeng Gao

备注：: 29 pages, 16 figures, technical report from MSR

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出多模态智能体基础模型 Magma，通过 SoM 与 ToM 标注的图像、视频和机器人数据预训练，获得时空规划与行动能力，并在 UI 导航、机器人操作等任务上达到最优。

摘要：

我们提出了 Magma，这是一个服务于数字世界和物理世界中多模态 AI 智能体任务的基础模型。Magma 是对视觉-语言（VL）模型的重要扩展：它不仅保留了后者的 VL 理解能力（言语智能），还具备在视觉-空间世界中进行规划和行动的能力（时空智能），并能够完成从 UI 导航到机器人操作的各类智能体任务。为赋予这些智能体能力，Magma 在涵盖图像、视频和机器人数据的大量异构数据集上进行预训练；其中，图像中可操作的视觉对象（例如 GUI 中可点击的按钮）通过 Set-of-Mark（SoM）进行标注，以实现动作落地，而视频中的对象运动（例如人手或机械臂的轨迹）通过 Trace-of-Mark（ToM）进行标注，以用于动作规划。大量实验表明，SoM 与 ToM 实现了很强的协同作用，并促进 Magma 模型获得时空智能；正如图 1 所示，这是广泛任务的基础。特别是，Magma 在 UI 导航和机器人操作任务上创造了新的最先进结果，优于此前专门针对这些任务定制的模型。在与图像和视频相关的多模态任务上，Magma 也表现得与那些在远更大数据集上训练的流行大型多模态模型相当甚至更好。为便于复现，我们在 https://microsoft.github.io/Magma 公开了模型和代码。

英文摘要：

We present Magma, a foundation model that serves multimodal AI agentic tasks in both the digital and physical worlds. Magma is a significant extension of vision-language (VL) models in that it not only retains the VL understanding ability (verbal intelligence) of the latter, but is also equipped with the ability to plan and act in the visual-spatial world (spatial-temporal intelligence) and complete agentic tasks ranging from UI navigation to robot manipulation. To endow the agentic capabilities, Magma is pretrained on large amounts of heterogeneous datasets spanning from images, videos to robotics data, where the actionable visual objects (e.g., clickable buttons in GUI) in images are labeled by Set-of-Mark (SoM) for action grounding, and the object movements (e.g., the trace of human hands or robotic arms) in videos are labeled by Trace-of-Mark (ToM) for action planning. Extensive experiments show that SoM and ToM reach great synergy and facilitate the acquisition of spatial-temporal intelligence for our Magma model, which is fundamental to a wide range of tasks as shown in Fig.1. In particular, Magma creates new state-of-the-art results on UI navigation and robotic manipulation tasks, outperforming previous models that are specifically tailored to these tasks. On image and video-related multimodal tasks, Magma also compares favorably to popular large multimodal models that are trained on much larger datasets. We make our model and code public for reproducibility at https://microsoft.github.io/Magma.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。