AMPLIFY：面向机器人视频学习的无动作先验运动框架
AMPLIFY: Actionless Motion Priors for Robot Learning from Videos
AI总结：
针对机器人带动作标签数据稀缺的问题，提出AMPLIFY模块化框架，通过无动作视频学习离散运动令牌构建动力学模型，在策略学习和视频预测等任务中实现性能提升与泛化突破。
摘要：
机器人领域带动作标签的数据稀缺且获取成本高昂，这限制了所学策略的泛化能力。相比之下，海量无动作视频数据唾手可得，但如何将这些观测结果转化为有效的策略仍是一大难题。我们提出了AMPLIFY这一全新框架，它通过将视觉动态编码为从关键点轨迹提取的紧凑离散运动令牌，来利用大规模视频数据。我们的模块化方法将视觉运动预测与动作推理分离开来，把“学习何种运动定义任务”和“机器人如何执行该运动”这两个难题解耦。我们在海量无动作视频上训练前向动力学模型，在少量带动作标签的样本上训练逆动力学模型，从而实现两者的独立扩展。大量评估实验表明，所学习的动力学模型不仅精度高——与现有方法相比，均方误差（MSE）最高降低3.7倍，像素预测准确率提升2.5倍以上——而且适用范围广。在下游策略学习中，我们的动力学预测在低数据场景下带来1.2-2.2倍的性能提升；通过无动作人类视频学习时，平均性能提升1.4倍；还首次实现了在零分布内动作数据的情况下向LIBERO任务的泛化。除机器人控制外，我们还发现AMPLIFY学习到的动力学模型是一种通用的潜世界模型，能够提升视频预测质量。我们的研究结果提出了一种利用异构数据源构建高效、可泛化世界模型的新范式。更多信息请访问https://amplify-robotics.github.io/。
英文摘要：
Action-labeled data for robotics is scarce and expensive, limiting the generalization of learned policies. In contrast, vast amounts of action-free video data are readily available, but translating these observations into effective policies remains a challenge. We introduce AMPLIFY, a novel framework that leverages large-scale video data by encoding visual dynamics into compact, discrete motion tokens derived from keypoint trajectories. Our modular approach separates visual motion prediction from action inference, decoupling the challenges of learning what motion defines a task from how robots can perform it. We train a forward dynamics model on abundant action-free videos and an inverse dynamics model on a limited set of action-labeled examples, allowing for independent scaling. Extensive evaluations demonstrate that the learned dynamics are both accurate, achieving up to 3.7x better MSE and over 2.5x better pixel prediction accuracy compared to prior approaches, and broadly useful. In downstream policy learning, our dynamics predictions enable a 1.2-2.2x improvement in low-data regimes, a 1.4x average improvement by learning from action-free human videos, and the first generalization to LIBERO tasks from zero in-distribution action data. Beyond robotic control, we find the dynamics learned by AMPLIFY to be a versatile latent world model, enhancing video prediction quality. Our results present a novel paradigm leveraging heterogeneous data sources to build efficient, generalizable world models. More information can be found at https://amplify-robotics.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
