KOROL：利用 Koopman 算子展开学习可可视化的物体特征以用于操作
KOROL: Learning Visualizable Object Feature with Koopman Operator Rollout for Manipulation
AI总结：
本文提出 KOROL，通过 Koopman 算子展开训练视觉特征提取器，在无需真值物体状态的情况下学习可解释物体特征，并在仿真与真实操作中优于 NDP 和 Diffusion Policy。
摘要：
由于物体与多指手之间交互所依赖的复杂非线性动力学，学习灵巧操作技能面临重大挑战。Koopman 算子已成为一种在线性框架内对此类非线性动力学建模的稳健方法。然而，现有方法依赖运行时访问真值（GT）物体状态，因此不适用于基于视觉的实际应用。不同于隐式学习用于控制的视觉特征的图像到动作策略，我们使用动力学模型（具体为 Koopman 算子）来学习对场景中机器人操作至关重要的、视觉上可解释的物体特征。我们利用特征提取器预测的物体特征构建 Koopman 算子，并用其以自回归方式推进系统状态。我们训练特征提取器，将场景信息嵌入物体特征，从而实现机器人轨迹的准确传播。我们在仿真和真实机器人任务上评估了该方法，结果表明，其性能分别是基于模型的模仿学习 NDP 的 1.08 倍和图像到动作 Diffusion Policy 的 1.16 倍。结果表明，我们的方法能够利用学习到的特征保持任务成功率，并将适用性扩展到没有 GT 物体状态的真实世界操作。项目视频和代码位于：https://github.com/hychen-naza/KOROL。
英文摘要：
Learning dexterous manipulation skills presents significant challenges due to complex nonlinear dynamics that underlie the interactions between objects and multi-fingered hands. Koopman operators have emerged as a robust method for modeling such nonlinear dynamics within a linear framework. However, current methods rely on runtime access to ground-truth (GT) object states, making them unsuitable for vision-based practical applications. Unlike image-to-action policies that implicitly learn visual features for control, we use a dynamics model, specifically the Koopman operator, to learn visually interpretable object features critical for robotic manipulation within a scene. We construct a Koopman operator using object features predicted by a feature extractor and utilize it to auto-regressively advance system states. We train the feature extractor to embed scene information into object features, thereby enabling the accurate propagation of robot trajectories. We evaluate our approach on simulated and real-world robot tasks, with results showing that it outperformed the model-based imitation learning NDP by 1.08$\times$ and the image-to-action Diffusion Policy by 1.16$\times$. The results suggest that our method maintains task success rates with learned features and extends applicability to real-world manipulation without GT object states. Project video and code are available at: \url{https://github.com/hychen-naza/KOROL}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
