arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.15058cs.CV
收藏

KETA:基于运动学短语增强的细粒度对齐文本到动作生成

KETA: Kinematic-Phrases-Enhanced Text-to-Motion Generation via Fine-grained Alignment

Zhiyuan College(致远学院);Shanghai Jiao Tong University(上海交通大学)

Yu Jiang, Yixing Chen, Xingyang Li

AI总结:

针对文本到动作生成中自然语言与物理域的鸿沟,提出 KETA 方法,利用运动学短语作为中间表示,通过文本分解与对齐器进行细粒度对齐,显著提升了生成动作的准确性与一致性。

摘要:

动作合成在人工智能的各个领域中发挥着重要作用。在动作生成的各种条件中,文本能够详尽地描述动作细节且易于获取,这使得文本到动作(T2M)生成显得尤为重要。最先进的 T2M 技术主要利用扩散模型,以文本提示作为引导来生成动作,从而解决 T2M 任务的多对多特性。然而,鉴于自然语言领域与物理领域之间存在鸿沟,现有的 T2M 方法面临挑战,难以生成与文本完全一致的动作为了解决这一问题,我们利用运动学短语(KP)这一连接两种模态的中间表示。我们提出的方法 KETA 通过语言模型将给定文本分解为若干子文本。该方法训练一个对齐器,将分解后的文本与从生成的动作中提取的 KP 片段进行对齐。从而能够限制基于扩散的 T2M 模型的行为。在训练阶段,我们将文本-KP 对齐损失作为辅助目标来监督模型。在推理阶段,我们在解码器结构中对生成的动作进行多轮优化,在每一轮新迭代中计算文本-KP 距离作为引导信号。实验表明,在基础模型和动作扩散模型这两种骨干网络上,KETA 的 R precision 和 FID 值分别提高了 1.19 倍和 2.34 倍。与广泛的 T2M 生成模型相比,KETA 取得了最佳或第二佳的性能。

英文摘要:

Motion synthesis plays a vital role in various fields of artificial intelligence. Among the various conditions of motion generation, text can describe motion details elaborately and is easy to acquire, making text-to-motion(T2M) generation important. State-of-the-art T2M techniques mainly leverage diffusion models to generate motions with text prompts as guidance, tackling the many-to-many nature of T2M tasks. However, existing T2M approaches face challenges, given the gap between the natural language domain and the physical domain, making it difficult to generate motions fully consistent with the texts. We leverage kinematic phrases(KP), an intermediate representation that bridges these two modalities, to solve this. Our proposed method, KETA, decomposes the given text into several decomposed texts via a language model. It trains an aligner to align decomposed texts with the KP segments extracted from the generated motions. Thus, it's possible to restrict the behaviors for diffusion-based T2M models. During the training stage, we deploy the text-KP alignment loss as an auxiliary goal to supervise the models. During the inference stage, we refine our generated motions for multiple rounds in our decoder structure, where we compute the text-KP distance as the guidance signal in each new round. Experiments demonstrate that KETA achieves up to 1.19x, 2.34x better R precision and FID value on both backbones of the base model, motion diffusion model. Compared to a wide range of T2M generation models. KETA achieves either the best or the second-best performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑