arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.21135cs.CVcs.GRcs.RO
收藏

Odoriko: 一种形状感知的多模态人体运动扩散框架

Odoriko: A Shape-Aware Multimodal Diffusion Framework for Human Motion

Sony Group Corporation(索尼集团公司);Sony AI(索尼人工智能)

Dongseok Shim, Julian Tanke, Kengo Uchida, Christian Simon, Koichi Saito, Takashi Shibuya, Shusuke Takahashi, Yuki Mitsufuji

AI总结:

提出首个统一的多模态运动生成框架Odoriko,通过扩散模型将主体生物形态信息融入运动合成,支持文本、音乐、视频输入,并在形态缺失时联合估计与生成。

摘要:

人体运动生成已在多种输入模态(文本、音乐和视频)中得到广泛研究,最近的工作已将这些模态统一到单一的多模态框架中。然而,尽管性别和体型等形态因素已知会产生不同的运动学特征,但现有的统一框架均未将其纳入生成过程,而是将所有主体视为形态等效。我们提出Odoriko,这是第一个反映主体生物形态信息并直接将其合成到运动输出中的统一多模态运动生成框架。Odoriko不是对主体变化进行平均,而是在单个模型内,针对文本、音乐和视频条件,生成与运动者身份一致的运动,而不仅仅是与要求执行的任务一致。当显式形态信息不可用时,Odoriko还能在生成运动的同时恢复主体形态,将估计和生成统一在一个框架中。在文本到运动、音乐到舞蹈和视频到运动基准上的大量实验表明,Odoriko在标准指标上达到或超过先前的专用模型,同时实现了现有统一框架无法支持的形态一致性生成。

英文摘要:

Human motion generation has been widely studied across diverse input modalities, text, music, and video, and recent efforts have unified these into single multimodal frameworks. However, while morphological factors such as gender and body shape are known to produce distinct kinematic signatures, no existing unified framework incorporates this into generation, treating all subjects as morphologically equivalent. We present Odoriko, the first unified multimodal motion generation framework that reflects subject bio-morphological information directly in synthesized motion output. Rather than averaging over subject variation, Odoriko generates motion that is consistent with who is moving, not just what they are asked to do, across text, music, and video conditions within a single model. When explicit morphological information is unavailable, Odoriko additionally recovers subject morphology alongside motion, unifying estimation and generation in one framework. Extensive experiments across text-to-motion, music-to-dance, and video-to-motion benchmarks demonstrate that Odoriko matches or exceeds prior specialized models on standard metrics, while enabling morphology-consistent generation that no existing unified framework supports.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑