MultiPLY：3D世界中以对象为中心的多感官具身大语言模型
MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World
AI总结：
本文提出MultiPLY，一个多感官具身大语言模型，通过构建50万条多感官交互数据集，将视觉、音频、触觉和温度信息与动作和状态token结合，在对象检索、工具使用等具身任务上显著超越基线。
摘要：
人类具备在主动探索和与3D世界交互的过程中融合多种感官线索的能力。然而，当前的多模态大语言模型被动地吸收感官数据作为输入，缺乏与3D环境中的对象主动交互并动态收集其多感官信息的能力。为了推动这一领域的研究，我们提出了MultiPLY，一个多感官具身大语言模型，能够将视觉、音频、触觉和温度等多感官交互数据整合到大语言模型中，从而建立词语、动作和感知之间的关联。为此，我们首先通过部署一个由大语言模型驱动的具身智能体与3D环境交互，收集了名为Multisensory Universe的大规模多感官交互数据集，包含50万条数据。为了在生成的数据上对预训练大语言模型进行指令微调，我们首先将3D场景编码为抽象的对象中心表示，然后引入动作token，表示具身智能体在环境中执行特定动作，以及状态token，表示智能体在每个时间步的多感官状态观测。在推理阶段，MultiPLY能够生成动作token，指示智能体在环境中执行动作并获得下一个多感官状态观测。随后，该观测通过状态token附加回大语言模型，以生成后续的文本或动作token。我们通过一系列多样化的具身任务——包括对象检索、工具使用、多感官描述和任务分解——证明了MultiPLY大幅优于基线方法。
英文摘要：
Human beings possess the capability to multiply a melange of multisensory cues while actively exploring and interacting with the 3D world. Current multi-modal large language models, however, passively absorb sensory data as inputs, lacking the capacity to actively interact with the objects in the 3D environment and dynamically collect their multisensory information. To usher in the study of this area, we propose MultiPLY, a multisensory embodied large language model that could incorporate multisensory interactive data, including visual, audio, tactile, and thermal information into large language models, thereby establishing the correlation among words, actions, and percepts. To this end, we first collect Multisensory Universe, a large-scale multisensory interaction dataset comprising 500k data by deploying an LLM-powered embodied agent to engage with the 3D environment. To perform instruction tuning with pre-trained LLM on such generated data, we first encode the 3D scene as abstracted object-centric representations and then introduce action tokens denoting that the embodied agent takes certain actions within the environment, as well as state tokens that represent the multisensory state observations of the agent at each time step. In the inference time, MultiPLY could generate action tokens, instructing the agent to take the action in the environment and obtain the next multisensory state observation. The observation is then appended back to the LLM via state tokens to generate subsequent text or action tokens. We demonstrate that MultiPLY outperforms baselines by a large margin through a diverse set of embodied tasks involving object retrieval, tool use, multisensory captioning, and task decomposition.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
