V-Dreamer：通过视频生成先验自动机器人仿真与轨迹合成
V-Dreamer: Automating Robotic Simulation and Trajectory Synthesis via Video Generation Priors
AI总结：
V-Dreamer通过视频生成先验自动构建仿真环境和可执行轨迹，利用大语言模型和3D生成模型生成物理合理的3D场景，并通过Sim-to-Gen模块实现高效轨迹合成。
摘要：
训练通用机器人需要大量多样化操作数据，但现实世界的数据收集成本过高，现有仿真器受限于固定资产库和手动启发式方法。为此，我们提出V-Dreamer，一个完全自动化的框架，能够从自然语言指令直接生成开放词汇的仿真准备操作环境和可执行专家轨迹。V-Dreamer采用新的生成管道，利用大语言模型和3D生成模型构建物理合理的3D场景，并通过几何约束验证确保稳定、无碰撞的布局。关键的是，对于行为合成，我们利用视频生成模型作为丰富的运动先验。这些视觉预测通过鲁棒的Sim-to-Gen视觉-运动学对齐模块（利用CoTracker3和VGGT）映射到可执行的机器人轨迹。该管道支持高视觉多样性和物理保真度，无需人工干预。为了评估生成的数据，我们在合成轨迹上训练模仿学习策略，涵盖多样化的物体和环境变化。在桌面上操作任务中使用Piper机械臂的广泛评估表明，我们的策略在仿真中能够稳健地泛化到未见物体，并实现有效的仿真到现实转移，成功操控新的现实物体。
英文摘要：
Training generalist robots demands large-scale, diverse manipulation data, yet real-world collection is prohibitively expensive, and existing simulators are often constrained by fixed asset libraries and manual heuristics. To bridge this gap, we present V-Dreamer, a fully automated framework that generates open-vocabulary, simulation-ready manipulation environments and executable expert trajectories directly from natural language instructions. V-Dreamer employs a novel generative pipeline that constructs physically grounded 3D scenes using large language models and 3D generative models, validated by geometric constraints to ensure stable, collision-free layouts. Crucially, for behavior synthesis, we leverage video generation models as rich motion priors. These visual predictions are then mapped into executable robot trajectories via a robust Sim-to-Gen visual-kinematic alignment module utilizing CoTracker3 and VGGT. This pipeline supports high visual diversity and physical fidelity without manual intervention. To evaluate the generated data, we train imitation learning policies on synthesized trajectories encompassing diverse object and environment variations. Extensive evaluations on tabletop manipulation tasks using the Piper robotic arm demonstrate that our policies robustly generalize to unseen objects in simulation and achieve effective sim-to-real transfer, successfully manipulating novel real-world objects.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
