AI 中文总结
DreamTraj利用含细粒度指令的MOVE数据集，从单张RGB图像和任务指令读取冻结视频扩散模型中间表示，直接解码6自由度物体轨迹，性能优于传统方法且速度提升4.6倍。
AI 中文摘要
操纵过程中物体轨迹的准确预测对于闭合感知-动作循环至关重要。现有进展受限于两方面：可用数据集缺乏细粒度语言到运动的标注，现有预测器要么依赖视频、深度、CAD模型等特权输入，要么通过成本高昂且易出错的感知管线从完全生成的视频中恢复运动。我们通过MOVE数据集缩小了监督差距，该数据集包含5038条以物体为中心的第一人称轨迹，每条轨迹都配有细粒度自然语言指令，而非粗略的动词-名词标签。我们进一步提出DreamTraj，该模型从单张RGB图像和任务指令中预测6自由度物体轨迹，推理时无需视频、深度或CAD模型；它不生成视频，而是在早期去噪步骤中从冻结的图像到视频扩散模型的内部表示中读取运动。一个轻量流匹配读取器将查询-键注意力轨迹和池化隐藏状态解码为相对6自由度位姿。据我们所知，这是首个直接从中间视频扩散表示而非生成像素中解码物体6自由度轨迹的方法。DreamTraj在平移和旋转预测任务上，相较于使用多帧或特权输入的预测器达到了新的最优水平，且运行速度比“生成后提取”管线快4.6倍。
英文摘要
Accurate prediction of object trajectories during manipulation is essential for closing the perception-action loop. Progress is limited on two fronts: available datasets lack fine-grained language-to-motion annotations, and existing predictors either rely on privileged inputs such as video, depth, or CAD models, or recover motion from fully generated videos through costly, error-prone perception pipelines. We close the supervision gap with the MOVE dataset, 5,038 object-centric egocentric trajectories, each paired with a fine-grained natural-language instruction rather than a coarse verb-noun label. We further propose DreamTraj, which predicts a 6-DoF object trajectory from a single RGB image and a task instruction, requiring no video, depth, or CAD model at inference: rather than generating a video, it reads motion from the internal representations of a frozen image-to-video diffusion model at an early denoising step. A lightweight flow-matching Reader decodes query-key attention tracks and pooled hidden states into relative 6-DoF poses. To our knowledge, this is the first approach to directly decode object 6-DoF trajectories from intermediate video diffusion representations rather than generated pixels. DreamTraj sets a new state of the art on both translation and rotation against forecasters that consume multi-frame or privileged inputs, and runs 4.6x faster than generate-then-extract pipelines.
Comments17 pages, 8 figures, 11 tables. Project page: https://whathappen0.github.io/DreamTraj/
