arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.07320cs.CV
收藏

CoMA:基于多模态智能体的组合式人体动作生成

CoMA: Compositional Human Motion Generation with Multi-modal Agents

University of California, Irvine(加利福尼亚大学欧文分校);Stony Brook University(石溪大学);Southeast University(东南大学);Chongqing University(重庆大学);Huazhong University of Science and Technology(华中科技大学);Northeastern University(东北大学)

Shanlin Sun, Gabriel De Araujo, Jiaqi Xu, Shenghan Zhou, Hanwen Zhang, Ziheng Huang, Chenyu You, Xiaohui Xie

AI总结:

本文提出 CoMA,通过协作式多模态大模型智能体与带部位编码器和码本的 mask transformer 生成器,实现复杂人体动作的生成、文本编辑、理解与自校正。

摘要:

近年来,3D人体动作生成取得了实质性进展。尽管最先进方法已显著提升性能,但它们仍难以处理训练数据中未见过的复杂且细致的动作,这主要源于动作数据集稀缺以及生成新训练样本的成本高昂。为应对这些挑战,我们提出 CoMA,这是一种用于复杂人体动作生成、编辑和理解的基于智能体的解决方案。CoMA 利用由大型语言模型和视觉模型驱动的多个协作智能体,并结合基于 mask transformer 的动作生成器,该生成器具备面向身体部位的编码器和码本,以实现细粒度控制。我们的框架能够依据详细指令生成短序列和长动作序列,支持文本引导的动作编辑,并通过自我修正提升质量。在 HumanML3D 数据集上的评估表明,其性能可与最先进方法相竞争。此外,我们构建了一组上下文丰富、具有组合性的长文本提示;用户研究显示,我们的方法显著优于现有方法。

英文摘要:

3D human motion generation has seen substantial advancement in recent years. While state-of-the-art approaches have improved performance significantly, they still struggle with complex and detailed motions unseen in training data, largely due to the scarcity of motion datasets and the prohibitive cost of generating new training examples. To address these challenges, we introduce CoMA, an agent-based solution for complex human motion generation, editing, and comprehension. CoMA leverages multiple collaborative agents powered by large language and vision models, alongside a mask transformer-based motion generator featuring body part-specific encoders and codebooks for fine-grained control. Our framework enables generation of both short and long motion sequences with detailed instructions, text-guided motion editing, and self-correction for improved quality. Evaluations on the HumanML3D dataset demonstrate competitive performance against state-of-the-art methods. Additionally, we create a set of context-rich, compositional, and long text prompts, where user studies show our method significantly outperforms existing approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑