超越画面：预测自聚焦三维视觉跨度
Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span
AI总结：
本文提出EgoSpanLift方法，通过将自聚焦视觉跨度预测从2D转换到3D，实现对三维环境未来视觉聚焦的高效预测，并构建了大规模基准测试集。
摘要：
人们基于驱动探索和行为的潜在意图持续感知和与周围环境互动。尽管关于自聚焦用户和场景理解的研究主要集中在运动和接触式交互上，但预测人类视觉感知本身仍较少被探索，尽管其在引导人类行为中的基础作用及其对AR/VR和辅助技术的影响不容忽视。我们针对自聚焦三维视觉跨度预测这一挑战，预测人在其三维环境中的下一次视觉聚焦位置。为此，我们提出了EgoSpanLift，一种将自聚焦视觉跨度预测从2D图像平面转换到3D场景的新方法。EgoSpanLift将SLAM导出的关键点转换为兼容目光的几何结构，并提取体积化的视觉跨度区域。我们进一步将EgoSpanLift与3D U-Net和单向变压器结合，实现时空融合，以高效预测未来视觉跨度。此外，我们从原始的自聚焦多感官数据中整理出一个全面的基准测试集，创建了一个包含364.6万样本的测试平台用于三维视觉跨度预测。我们的方法在自聚焦2D目光预测和3D定位方面优于竞争基线，即使在投影回2D图像平面且不进行额外2D特定训练时也能够取得可比的结果。
英文摘要：
People continuously perceive and interact with their surroundings based on underlying intentions that drive their exploration and behaviors. While research in egocentric user and scene understanding has focused primarily on motion and contact-based interaction, forecasting human visual perception itself remains less explored despite its fundamental role in guiding human actions and its implications for AR/VR and assistive technologies. We address the challenge of egocentric 3D visual span forecasting, predicting where a person's visual perception will focus next within their three-dimensional environment. To this end, we propose EgoSpanLift, a novel method that transforms egocentric visual span forecasting from 2D image planes to 3D scenes. EgoSpanLift converts SLAM-derived keypoints into gaze-compatible geometry and extracts volumetric visual span regions. We further combine EgoSpanLift with 3D U-Net and unidirectional transformers, enabling spatio-temporal fusion to efficiently predict future visual span in the 3D grid. In addition, we curate a comprehensive benchmark from raw egocentric multisensory data, creating a testbed with 364.6K samples for 3D visual span forecasting. Our approach outperforms competitive baselines for egocentric 2D gaze anticipation and 3D localization while achieving comparable results even when projected back onto 2D image planes without additional 2D-specific training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
