Timo: 驯服多模态扩散Transformer用于人体运动生成

中文摘要 AI 辅助

大多数现有的人体运动生成（HMG）方法使用交叉注意力模块来注入文本语义，但忽略了运动和文本标记之间双向建模的重要性，这限制了文本理解。一个直接的想法是将多模态扩散Transformer（MMDiT）引入HMG，该模型在视觉生成中已显示出有效的联合文本-视觉建模。然而，我们发现关节运动在时间上是连贯的，但关节间相关性较弱，直接应用带有流匹配的MMDiT会产生协调性差且生硬的运动。在这项工作中，我们提出了Timo，一种专为HMG定制的新型运动学感知MMDiT框架。Timo结合了完全共享的多模态注意力，用于双向文本-运动建模，并采用流匹配、几何和旋转运动学监督（比较实际旋转及其随时间的变化），以及一个从广泛运动学习到详细字幕对齐的两阶段课程。此外，我们构建了一个包含来自六个公共数据集的40,025个保留剪辑的基准，涵盖多种动作，在统一的评估器和评分协议下评估六个互补维度。我们的模型在定量和定性评估中均显著优于最先进的方法。值得注意的是，Timo在六个维度中的五个上超越了Kimodo，平均基准分数相对提高了40.8%。项目页面：此https URL。演示页面：此https URL。

英文摘要

Most existing human motion generation (HMG) methods use cross-attention modules to inject text semantics, but ignore the importance of bidirectional modeling between motion and text tokens, which limits text comprehension. A straightforward idea is introducing multimodal diffusion transformers (MMDiT), which have shown effective joint text--visual modeling in vision generation, into HMG. However, we find that articulated motion is temporally coherent but weakly correlated across joints, in which directly applying an MMDiT with flow matching produces poorly coordinated and jerky motion. In this work, we propose Timo, a novel kinematics-aware MMDiT framework tailored for HMG. Timo combines fully shared multimodal attention for bidirectional text--motion modeling with flow matching, geometric and rotational-kinematics supervision that compares actual rotations and their changes over time, and a two-stage curriculum progressing from broad motion learning to detailed caption alignment. Further, we construct a benchmark of $40{,}025$ held-out clips from six public datasets spanning diverse actions, assessing six complementary dimensions under a common evaluator and scoring protocol. Our model substantially outperforms state-of-the-art methods in both quantitative and qualitative evaluations. Remarkably, Timo surpasses Kimodo on five of six dimensions, achieving a $40.8$% relative improvement in the average benchmark score. Project page: https://kyfafyd.wang/projects/timo. Demo page: https://timo.kyfafyd.wang.

发表机构

• LimX Dynamics

机构由 AI 辅助整理，请以论文原文为准。