EgoExo-Gen：通过观察外向视角视频实现内向视角视频预测
EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos
AI总结：
针对跨视角视频预测任务，提出EgoExo-Gen模型，通过明确建模手-物体动态（两阶段：跨视角HOI掩码预测+带结构引导的视频扩散），在Ego-Exo4D和H2O数据集上性能优于现有模型，显著提升内向视角视频中手和交互物体的生成质量。
摘要：
在第一人称视角下生成视频在增强现实和具身智能领域具有广阔的应用前景。本研究探索跨视角视频预测任务：给定一段外向视角视频、对应内向视角视频的首帧以及文本指令，目标是生成内向视角视频的未来帧。受内向视角视频中的手-物体交互（HOI）代表当前行为者主要意图和动作这一观点启发，我们提出EgoExo-Gen，其明确建模手-物体动态以实现跨视角视频预测。EgoExo-Gen包含两个阶段：首先，设计跨视角HOI掩码预测模型，通过建模时空内向-外向对应关系预测未来内向帧中的HOI掩码；其次，采用视频扩散模型，利用内向首帧和文本指令预测未来内向帧，同时将HOI掩码作为结构引导以提升预测质量。为便于训练，我们开发自动化流水线，利用视觉基础模型为内向和外向视频生成伪HOI掩码。大量实验表明，在Ego-Exo4D和H2O基准数据集上，我们提出的EgoExo-Gen相较于现有视频预测模型取得更优的预测性能，HOI掩码显著改善了内向视角视频中手部和交互物体的生成效果。
英文摘要：
Generating videos in the first-person perspective has broad application prospects in the field of augmented reality and embodied intelligence. In this work, we explore the cross-view video prediction task, where given an exo-centric video, the first frame of the corresponding ego-centric video, and textual instructions, the goal is to generate futur frames of the ego-centric video. Inspired by the notion that hand-object interactions (HOI) in ego-centric videos represent the primary intentions and actions of the current actor, we present EgoExo-Gen that explicitly models the hand-object dynamics for cross-view video prediction. EgoExo-Gen consists of two stages. First, we design a cross-view HOI mask prediction model that anticipates the HOI masks in future ego-frames by modeling the spatio-temporal ego-exo correspondence. Next, we employ a video diffusion model to predict future ego-frames using the first ego-frame and textual instructions, while incorporating the HOI masks as structural guidance to enhance prediction quality. To facilitate training, we develop an automated pipeline to generate pseudo HOI masks for both ego- and exo-videos by exploiting vision foundation models. Extensive experiments demonstrate that our proposed EgoExo-Gen achieves better prediction performance compared to previous video prediction models on the Ego-Exo4D and H2O benchmark datasets, with the HOI masks significantly improving the generation of hands and interactive objects in the ego-centric videos.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
