在关键步骤级别学习人类技能生成器
Learning Human Skill Generators at Key-Step Levels
AI总结：
针对现有视频生成模型难以处理人类技能复杂流程的问题，提出关键步骤技能生成（KS-Gen）任务及框架，通过MLLM生成关键步骤描述、KIG解决步骤不连续性并生成高时间一致性视频片段，促进技能学习与具身智能发展。
摘要：
我们致力于在关键步骤级别学习人类技能生成器。技能生成是一项具有挑战性的工作，但其成功实现可极大促进人类技能学习，并为具身智能提供更多经验。尽管当前视频生成模型能合成简单、原子化的人类操作，但因人类技能的复杂流程，它们在处理人类技能时面临困难。人类技能涉及多步骤、长时间动作及复杂场景转换，因此现有合成长视频的朴素自回归方法无法生成人类技能。为此，我们提出一项新任务——关键步骤技能生成（KS-Gen），旨在降低生成人类技能视频的复杂度。给定初始状态和技能描述，该任务需生成完成技能的关键步骤视频片段，而非完整视频。为支持此任务，我们引入精心构建的数据集，并定义多种评估指标以评估性能。考虑到KS-Gen的复杂性，我们提出新框架：首先，多模态大型语言模型（MLLM）利用检索参数生成关键步骤描述；随后，关键步骤图像生成器（KIG）解决技能视频中关键步骤间的不连续性；最后，视频生成模型利用这些描述和关键步骤图像生成具有高时间一致性的关键步骤视频片段。我们对结果进行详细分析，希望为人类技能生成提供更多见解。所有模型和数据可在https://github.com/MCG-NJU/KS-Gen获取。
英文摘要：
We are committed to learning human skill generators at key-step levels. The generation of skills is a challenging endeavor, but its successful implementation could greatly facilitate human skill learning and provide more experience for embodied intelligence. Although current video generation models can synthesis simple and atomic human operations, they struggle with human skills due to their complex procedure process. Human skills involve multi-step, long-duration actions and complex scene transitions, so the existing naive auto-regressive methods for synthesizing long videos cannot generate human skills. To address this, we propose a novel task, the Key-step Skill Generation (KS-Gen), aimed at reducing the complexity of generating human skill videos. Given the initial state and a skill description, the task is to generate video clips of key steps to complete the skill, rather than a full-length video. To support this task, we introduce a carefully curated dataset and define multiple evaluation metrics to assess performance. Considering the complexity of KS-Gen, we propose a new framework for this task. First, a multimodal large language model (MLLM) generates descriptions for key steps using retrieval argument. Subsequently, we use a Key-step Image Generator (KIG) to address the discontinuity between key steps in skill videos. Finally, a video generation model uses these descriptions and key-step images to generate video clips of the key steps with high temporal consistency. We offer a detailed analysis of the results, hoping to provide more insights on human skill generation. All models and data are available at https://github.com/MCG-NJU/KS-Gen.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
