MuseVLA: 一种用于机器人操作的自适应多模态感知视觉-语言-动作模型
MuseVLA: An Adaptive Multimodal Sensing Vision-Language-Action Model for Robotic Manipulation
AI总结：
提出MuseVLA模型，通过将传感器作为按需工具集成，实现自适应多模态感知；设计传感器图像统一表示，并引入数据合成流水线，在灵巧手操作任务中平均成功率80.6%，显著优于RGB-only和多模态基线。
摘要：
人类自然地利用多种感知模态与物理世界交互，而大多数用于机器人的视觉-语言-动作（VLA）模型仅依赖RGB观测。这限制了它们感知难以或无法从RGB相机推断的物理属性（如温度、声音或雷达响应）的能力。我们提出MuseVLA，一种自适应多模态感知VLA模型，将新型传感器作为按需工具集成到机器人操作中。给定任务指令和视觉上下文，MuseVLA首先生成一个传感器令牌和目标描述，选择要调用的感知模态和关注对象，类似于带参数的工具调用。然后，它将选定的传感器测量值转换为接地传感器图像，这是一种统一的中间表示，编码异构读数以进行多模态融合和动作生成。这种设计将传感器特定处理与VLA主干解耦，实现了多种模态的高效集成。为了减少对昂贵的多传感器机器人数据集的需求，我们进一步引入了一种数据合成流水线，用接地传感器图像增强现有的RGB视频数据集，从而实现对未见过的传感器引导任务的泛化。我们在真实机器人上评估了MuseVLA，涉及需要多模态感知输入的挑战性灵巧手操作任务，包括温度引导的拾取与放置、音频驱动的物体搜索和雷达辅助的隐藏物体检索。MuseVLA平均成功率达到80.6%，显著优于仅RGB和多模态VLA基线，并在未见任务上表现出强大的零样本能力。
英文摘要：
Humans naturally leverage diverse sensing modalities to interact with the physical world, while most Vision-Language-Action (VLA) models for robotics rely solely on RGB observations. This limits their ability to perceive physical properties that are difficult or impossible to infer from RGB cameras, such as temperature, sound, or radar response. We present MuseVLA, an adaptive multimodal sensing VLA model that integrates novel sensors as on-demand tools for robotic manipulation. Given a task instruction and visual context, MuseVLA first generates a sensor token and target description that select the sensing modality to invoke and what to attend to, analogous to a tool call with arguments. It then converts the selected sensor measurement into a grounded sensor image, a unified intermediate representation that encodes heterogeneous readings for multimodal fusion and action generation. This design decouples sensor-specific processing from the VLA backbone, enabling efficient integration of diverse modalities. To reduce the need for expensive multisensory robot datasets, we further introduce a data synthesis pipeline that augments existing RGB video datasets with grounded sensor images, enabling generalization to unseen sensor-guided tasks. We evaluate MuseVLA on a real-world robot across challenging dexterous hand manipulation tasks that require multimodal sensing inputs, including temperature-guided pick-and-place, audio-driven object search, and radar-assisted hidden object retrieval. MuseVLA achieves 80.6% success rate on average, outperforming RGB-only and multisensory VLA baselines significantly, and exhibits strong zero-shot capabilities on unseen tasks. Code, model and dataset are available at https://github.com/microsoft/MuseVLA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
