Aria-NeRF：多模态第一人称视角视图合成
Aria-NeRF: Multimodal Egocentric View Synthesis
AI总结：
本文提出Aria-NeRF，利用Meta Aria眼镜采集的多模态第一人称数据构建类NeRF场景模型，以加速多模态场景建模研究，推动VR/AR与机器人应用。
摘要：
我们旨在加速基于第一人称视角数据训练丰富多模态场景模型的研究，该研究基于受神经辐射场（NeRF）启发的可微分体积光线追踪技术。从第一人称视角图像序列构建类NeRF模型，在理解人类行为方面发挥着关键作用，并在VR/AR领域具有多种应用。此类第一人称视角的类NeRF模型可用作真实感仿真，显著推动能够在现实世界中执行任务的智能体的发展。第一人称视角视图合成的未来可能超越当今的NeRF，通过将视觉数据与多模态传感器增强，例如用于自我运动追踪的IMU、用于捕捉表面纹理和人类语言上下文的音频传感器，以及用于推断场景中人类注意力模式的眼动追踪器，从而产生新颖的环境表示。为支持和促进第一人称视角多模态场景建模的开发与评估，我们提出了一个全面的多模态第一人称视角视频数据集。该数据集提供了全面的传感数据集合，包括RGB图像、眼动追踪摄像头画面、来自麦克风的音频记录、来自气压计的大气压力读数、来自GPS的位置坐标、来自Wi-Fi和蓝牙的连接细节，以及与磁力计配对的双频IMU数据集（1kHz和800Hz）的信息。该数据集使用Meta Aria Glasses可穿戴设备平台采集。数据集中捕获的多样数据模态和真实世界背景，为加深我们对人类行为的理解，并在VR、AR和机器人领域实现更沉浸和智能的体验，奠定了坚实的基础。
英文摘要：
We seek to accelerate research in developing rich, multimodal scene models trained from egocentric data, based on differentiable volumetric ray-tracing inspired by Neural Radiance Fields (NeRFs). The construction of a NeRF-like model from an egocentric image sequence plays a pivotal role in understanding human behavior and holds diverse applications within the realms of VR/AR. Such egocentric NeRF-like models may be used as realistic simulations, contributing significantly to the advancement of intelligent agents capable of executing tasks in the real-world. The future of egocentric view synthesis may lead to novel environment representations going beyond today's NeRFs by augmenting visual data with multimodal sensors such as IMU for egomotion tracking, audio sensors to capture surface texture and human language context, and eye-gaze trackers to infer human attention patterns in the scene. To support and facilitate the development and evaluation of egocentric multimodal scene modeling, we present a comprehensive multimodal egocentric video dataset. This dataset offers a comprehensive collection of sensory data, featuring RGB images, eye-tracking camera footage, audio recordings from a microphone, atmospheric pressure readings from a barometer, positional coordinates from GPS, connectivity details from Wi-Fi and Bluetooth, and information from dual-frequency IMU datasets (1kHz and 800Hz) paired with a magnetometer. The dataset was collected with the Meta Aria Glasses wearable device platform. The diverse data modalities and the real-world context captured within this dataset serve as a robust foundation for furthering our understanding of human behavior and enabling more immersive and intelligent experiences in the realms of VR, AR, and robotics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
