以手控世界：基于手部动作的世界操控
Controlling the World by Sleight of Hand
AI总结：
针对现有生成模型无法基于动作条件进行物体操控的问题，本研究提出CosHand模型，通过学习海量人手与物体交互的无标注视频训练动作条件生成模型，可预测手物交互效果、建模不确定性，且能泛化到机器人手等不同具身形态。
摘要：
人类会自然地构建物体交互与动力学的心智模型，从而能够想象如果采取特定行动，周围环境会如何变化。尽管当前的生成模型在无条件生成/编辑图像或基于文本条件生成/编辑图像方面已展现出令人印象深刻的成果，但现有方法尚不具备基于动作条件执行物体操控的能力，而这是世界建模与动作规划的重要工具。为此，我们提出通过学习人类手部与物体交互的无标注视频，来训练动作条件生成模型。互联网上存在海量此类数据，可实现高效的规模扩展，从而支撑高性能动作条件模型的训练。给定一张图像以及期望的手部交互的形状/位置，CosHand（手部交互生成模型）能够合成交互发生后的未来图像。实验表明，所得模型可以很好地预测手-物交互的效果，尤其在未见环境中对未见物体的平移、拉伸和挤压交互上具备强泛化性。此外，可对CosHand进行多次采样以预测多种可能的效果，从而对交互/环境中的力的不确定性进行建模。最后，该方法可泛化至不同的具身形态，包括非人类的手，即机器人手，这表明生成式视频模型可成为机器人领域的强大模型。
英文摘要：
Humans naturally build mental models of object interactions and dynamics, allowing them to imagine how their surroundings will change if they take a certain action. While generative models today have shown impressive results on generating/editing images unconditionally or conditioned on text, current methods do not provide the ability to perform object manipulation conditioned on actions, an important tool for world modeling and action planning. Therefore, we propose to learn an action-conditional generative models by learning from unlabeled videos of human hands interacting with objects. The vast quantity of such data on the internet allows for efficient scaling which can enable high-performing action-conditional models. Given an image, and the shape/location of a desired hand interaction, CosHand, synthesizes an image of a future after the interaction has occurred. Experiments show that the resulting model can predict the effects of hand-object interactions well, with strong generalization particularly to translation, stretching, and squeezing interactions of unseen objects in unseen environments. Further, CosHand can be sampled many times to predict multiple possible effects, modeling the uncertainty of forces in the interaction/environment. Finally, method generalizes to different embodiments, including non-human hands, i.e. robot hands, suggesting that generative video models can be powerful models for robotics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
