SPA：3D空间感知实现有效的具身表示
SPA: 3D Spatial-Awareness Enables Effective Embodied Representation
AI总结：
本文提出SPA框架，通过可微神经渲染赋予ViT 3D空间理解能力，在268项任务中优于10多种先进方法，证实3D空间感知对具身表示学习的关键作用。
摘要：
在本文中，我们介绍了SPA，一种新颖的表示学习框架，强调3D空间感知在具身AI中的重要性。我们的方法在多视图图像上利用可微神经渲染，赋予原始Vision Transformer（ViT）固有的空间理解能力。我们进行了迄今为止最全面的具身表示学习评估，涵盖8个模拟器中的268项任务，包括单任务和语言条件多任务场景中的多种策略。结果令人信服：SPA在使用更少训练数据的同时，持续优于10多种最先进的表示方法，包括专为具身AI、视觉中心任务和多模态应用设计的方法。此外，我们进行了一系列真实世界实验，以确认其在实际场景中的有效性。这些结果突出了3D空间感知对具身表示学习的关键作用。我们最强的模型训练耗时超过6000个GPU小时，我们承诺开源所有代码和模型权重，以促进具身表示学习的未来研究。项目页面：https://haoyizhu.github.io/spa/。
英文摘要：
In this paper, we introduce SPA, a novel representation learning framework that emphasizes the importance of 3D spatial awareness in embodied AI. Our approach leverages differentiable neural rendering on multi-view images to endow a vanilla Vision Transformer (ViT) with intrinsic spatial understanding. We present the most comprehensive evaluation of embodied representation learning to date, covering 268 tasks across 8 simulators with diverse policies in both single-task and language-conditioned multi-task scenarios. The results are compelling: SPA consistently outperforms more than 10 state-of-the-art representation methods, including those specifically designed for embodied AI, vision-centric tasks, and multi-modal applications, while using less training data. Furthermore, we conduct a series of real-world experiments to confirm its effectiveness in practical scenarios. These results highlight the critical role of 3D spatial awareness for embodied representation learning. Our strongest model takes more than 6000 GPU hours to train and we are committed to open-sourcing all code and model weights to foster future research in embodied representation learning. Project Page: https://haoyizhu.github.io/spa/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
