MotIF：运动指令微调
MotIF: Motion Instruction Fine-tuning
AI总结：
针对VLMs难以基于完整轨迹判断机器人任务成功的问题，提出MotIF方法，通过轨迹级抽象表示微调VLMs，引入MotIF-1K数据集，模型精度至少提升两倍、召回率提升56.1%，可泛化且支持规划优化与轨迹排序。
摘要：
虽然许多机器人任务的成功仅通过观察最终状态与初始状态的差异即可判定——例如苹果是否被拿起——但很多任务需要观察机器人的完整运动才能正确判断成功。例如，梳头需要与头发轮廓和类型相匹配的重复动作。现有研究常使用现成的视觉语言模型（VLMs）作为成功检测器；然而，当成功取决于完整轨迹时，VLMs难以做出正确判断，原因有二：首先，现代VLMs仅在单帧上训练，无法捕捉完整轨迹的变化；其次，即使向最先进的VLMs提供多帧聚合输入，它们仍因缺乏机器人数据而无法检测成功。我们的核心思路是使用能捕捉轨迹级信息（如机器人运动路径）的抽象表示微调VLMs，具体方法为运动指令微调（MotIF），该方法通过上述抽象表示将机器人行为与环境进行语义关联。为基准测试和微调用于机器人运动理解的VLMs，我们引入MotIF-1K数据集，包含13个任务类别下的653条人类演示和369条机器人演示。MotIF根据轨迹的图像观察、任务指令和运动描述评估机器人运动的成功性。我们的模型在精度上比最先进的VLMs至少高出两倍，召回率高出56.1%，且在未见过的运动、任务和环境中具有泛化能力。最后，我们展示了MotIF在优化和终止机器人规划、根据轨迹与任务及运动描述的对齐度进行排序方面的实际应用。项目页面：https://motif-1k.github.io
英文摘要：
While success in many robotics tasks can be determined by only observing the final state and how it differs from the initial state - e.g., if an apple is picked up - many tasks require observing the full motion of the robot to correctly determine success. For example, brushing hair requires repeated strokes that correspond to the contours and type of hair. Prior works often use off-the-shelf vision-language models (VLMs) as success detectors; however, when success depends on the full trajectory, VLMs struggle to make correct judgments for two reasons. First, modern VLMs are trained only on single frames, and cannot capture changes over a full trajectory. Second, even if we provide state-of-the-art VLMs with an aggregate input of multiple frames, they still fail to detect success due to a lack of robot data. Our key idea is to fine-tune VLMs using abstract representations that are able to capture trajectory-level information such as the path the robot takes by overlaying keypoint trajectories on the final image. We propose motion instruction fine-tuning (MotIF), a method that fine-tunes VLMs using the aforementioned abstract representations to semantically ground the robot's behavior in the environment. To benchmark and fine-tune VLMs for robotic motion understanding, we introduce the MotIF-1K dataset containing 653 human and 369 robot demonstrations across 13 task categories. MotIF assesses the success of robot motion given the image observation of the trajectory, task instruction, and motion description. Our model significantly outperforms state-of-the-art VLMs by at least twice in precision and 56.1% in recall, generalizing across unseen motions, tasks, and environments. Finally, we demonstrate practical applications of MotIF in refining and terminating robot planning, and ranking trajectories on how they align with task and motion descriptions. Project page: https://motif-1k.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
