发表机构
The University of Texas at Austin; University of Florida; BrainCo（德克萨斯大学奥斯汀分校; 佛罗里达大学; BrainCo）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
DEXTERA提出自动化真实到仿真到真实框架，从单张RGB图像生成可部署灵巧操作策略，通过四阶段流程和联合训练将成功率从29.2%提升至61.9%。
AI 中文摘要
收集真实世界的机器人灵巧操作数据既昂贵又耗时。尽管高保真物理仿真器能够支持可扩展的数据合成和策略学习，但手动构建可部署的数字孪生仍然劳动密集，且残余的视觉、几何和动力学差距阻碍了可靠的仿真到真实迁移。我们提出DEXTERA，一个自动化的真实到仿真到真实框架，将单张RGB图像转化为可部署的灵巧操作策略，涵盖四个统一阶段：（1）单图像场景分解为静态高斯背景和具有VLM推断物理参数的可交互刚性或铰接资产；（2）度量场景全局对齐、对象规范化和形态平衡的机器人校准；（3）可扩展的仿真器任务原语构建、VR遥操作和以对象为中心的轨迹合成；（4）支持模仿学习和强化学习的共享多模态策略接口。我们在13个任务-具身组合、2个灵巧机器人平台和6种策略架构上评估DEXTERA。实验结果表明，与生成式基线相比，DEXTERA实现了优越的视觉保真度和3D几何重建，而跨域轨迹重放验证了强物理交互一致性。此外，仅仿真训练的策略能够实现可行的零样本真实机器人部署，而仿真-真实联合训练将不同策略架构的平均物理策略成功率从29.2%显著提升至61.9%。
英文摘要
Collecting real-world robot data for dexterous manipulation is costly and time-consuming. While high-fidelity physics simulators enable scalable data synthesis and policy learning, constructing deployment-ready digital twins manually remains labor-intensive, and residual visual, geometric, and dynamics gaps hinder reliable sim-to-real transfer. We present DEXTERA, an automated real-to-sim-to-real framework that transforms a single RGB image into deployable policies for dexterous manipulation across four unified stages: (1) single-image scene factorization into a static Gaussian background and interactive rigid or articulated assets with VLM-inferred physical parameters; (2) metric scene global alignment, object canonicalization, and morphology-balanced robot calibration; (3) scalable simulator task primitive construction, VR teleoperation, and object-centric trajectory synthesis; and (4) a shared multimodal policy interface supporting both imitation learning and reinforcement learning. We evaluate DEXTERA across 13 task-embodiment pairs, 2 dexterous robot platforms, and 6 policy architectures. Experimental results demonstrate that DEXTERA achieves superior visual fidelity and 3D geometric reconstruction compared to generative baselines, while cross-domain trajectory replays validate strong physical interaction consistency. Furthermore, simulation-only trained policies enable viable zero-shot real-robot deployment, while simulation-real co-training substantially improves mean physical policy success from 29.2% to 61.9% across diverse policy architectures. Project website: https://dextera-project.github.io/
