MotionCraft：使用即插即用多模态控制生成全身运动
MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controls
AI总结：
提出统一扩散Transformer框架MotionCraft，通过从粗到细的训练策略和MC-Attn并行建模人体拓扑图，实现即插即用的多模态全身运动生成，并构建首个统一格式的基准MC-Bench，在多项任务上达到最优性能。
摘要：
由文本、语音或音乐控制的全身多模态运动生成在视频生成和角色动画等领域有众多应用。然而，采用统一模型来实现具有不同条件模态的各种生成任务面临两大主要挑战：不同任务间（如语音伴随手势和文本驱动的日常动作）的运动分布漂移，以及不同粒度混合条件（如文本和音频）的复杂优化。此外，不同任务和数据集间不一致的运动格式阻碍了面向多模态运动生成的有效训练。本文提出MotionCraft，一种统一的扩散Transformer，通过即插即用的多模态控制来生成全身运动。该框架采用从粗到细的训练策略，首先进行文本到运动的语义预训练，随后进行多模态低层控制适应，以处理不同粒度的条件。为有效学习和迁移不同分布间的运动知识，我们设计了MC-Attn，用于对静态和动态人体拓扑图进行并行建模。为克服现有基准的运动格式不一致问题，我们引入了MC-Bench，这是首个基于统一SMPL-X格式的可用多模态全身运动生成基准。大量实验表明，MotionCraft在各种标准运动生成任务上达到了最先进的性能。
英文摘要：
Whole-body multimodal motion generation, controlled by text, speech, or music, has numerous applications including video generation and character animation. However, employing a unified model to achieve various generation tasks with different condition modalities presents two main challenges: motion distribution drifts across different tasks (e.g., co-speech gestures and text-driven daily actions) and the complex optimization of mixed conditions with varying granularities (e.g., text and audio). Additionally, inconsistent motion formats across different tasks and datasets hinder effective training toward multimodal motion generation. In this paper, we propose MotionCraft, a unified diffusion transformer that crafts whole-body motion with plug-and-play multimodal control. Our framework employs a coarse-to-fine training strategy, starting with the first stage of text-to-motion semantic pre-training, followed by the second stage of multimodal low-level control adaptation to handle conditions of varying granularities. To effectively learn and transfer motion knowledge across different distributions, we design MC-Attn for parallel modeling of static and dynamic human topology graphs. To overcome the motion format inconsistency of existing benchmarks, we introduce MC-Bench, the first available multimodal whole-body motion generation benchmark based on the unified SMPL-X format. Extensive experiments show that MotionCraft achieves state-of-the-art performance on various standard motion generation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
