感知与预测动态运动:用于第一人称视角网球视频的视觉模型中的时空智能
Perceiving and Predicting Dynamic Sports: Spatiotemporal Intelligence in Vision Models for Egocentric Tennis Video
AI总结:
该研究构建了带多模态标注的网球第一人称视角数据集,评估了多种视觉模型的时空预测性能,提出条件扩散模型生成网球运动,为动态机器人的视觉时空智能提供了支撑。
摘要:
敏捷机器人需要时空智能来解释场景、推理交互并预测未来事件。然而,目前尚不清楚基础视觉模型在多大程度上支持这些能力,尤其是从移动的第一人称视角相机(这是机器人的自然视角)出发的情况。为了研究这一点,我们收集了带标注的网球数据集,并在时空任务上训练了一组视觉模型。我们的数据集包含与5个第三人称视角视频同步的第一人称视角视频、估计的全局位置、标注的击球时间以及标注的击球类型。随后,我们评估了从第一人称视角图像出发的7项预测任务:2D自身球员定位与3D球定位、未来球员与球的轨迹、正手或反手击球预测、到下一次自身球员-球接触的时间、以及该接触时刻的3D球位置。我们比较了图像编码器、视频编码器和视觉-语言模型,这些模型均配有联合训练的预测头。表现最佳的模型实现了当前帧的平均定位误差:自身球员为0.14米,球为0.54米;1.1秒内的未来预测误差:自身球员为0.34米,球为0.91米;平均接触时间预测误差为0.14秒。辅助深度信息显著提升了3D球定位效果,这表明推断度量深度仍是一项挑战。更长的图像序列并未改善球的预测,说明模型表征利用了场景上下文,而非仅依赖帧间运动。最后,我们提出了一种条件扩散模型,用于从第一人称视角视频的视觉编码器预测中实时生成全身网球运动学动作,并在预测优化时调整动作。这些结果推动了对基础视觉模型如何利用第一人称视角视频支持实时动态机器人的更深入理解。
英文摘要:
Agile robots require spatiotemporal intelligence to interpret scenes, reason about interactions, and anticipate future events. Yet it remains unclear how well foundation vision models support these capabilities, particularly from a moving egocentric camera, which is a natural view for robots. To investigate this, we collect an annotated tennis dataset and train a suite of vision models on spatiotemporal tasks. Our dataset contains egocentric video synchronized with five allocentric videos, estimated global positions, labeled contact times, and annotated stroke types. We then evaluate seven prediction tasks from egocentric images: 2D ego player and 3D ball localization, future player and ball trajectories, forehand or backhand stroke anticipation, time to the next ego player ball contact, and 3D ball position at that contact. We compare image encoders, video encoders, and vision-language models with jointly trained prediction heads. The best performing models achieve mean current-frame localization errors of 0.14 m for the ego player and 0.54 m for the ball, future forecasting errors of 0.34 m for the ego player and 0.91 m for the ball over 1.1 s, and a mean contact-time prediction error of 0.14 s. Auxiliary depth information substantially improves 3D ball localization, suggesting that inferring metric depth remains a challenge. Longer image histories do not improve ball forecasting, indicating that representations exploit scene context rather than rely solely on inter-frame motion. Finally, we propose a conditional diffusion model to generate whole-body kinematic tennis motion from vision encoder predictions on egocentric video at real-time rates, adapting movements as predictions are refined. These results move towards a deeper understanding of how foundation vision models can support real-time dynamic robots using egocentric video.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。