arXiv 2408.093972024-08cs.CV

Combo：和谐统一的协同语音整体3D人体运动生成与高效可定制适配

Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony

Wolf 1069B Lab, Sany Group（沃尔夫1069B实验室，三一集团）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Engineering and Technology, University of Washington, Tacoma（华盛顿大学塔科姆分校工程与科技学院）；Alibaba Group（阿里巴巴集团）；Department of Engineering, King’s College London（伦敦大学国王学院工程系）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）

Chao Xu, Mingze Sun, Zhi-Qi Cheng, Fei Wang, Yang Liu, Baigui Sun, Ruqi Huang, Alexander Hauptmann

备注：: Accepted to TPAMI

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 Combo 框架，通过预训练与 X-Adapter 参数高效微调解耦语音和角色条件，并利用 DU-Trans 分支联合建模面部与身体运动，实现高质量协同语音整体3D人体运动生成及高效身份与情感适配。

摘要：

本文提出了一个新颖的框架 Combo，用于和谐的协同语音整体3D人体运动生成和高效的可定制适配。我们特别指出，一个根本性挑战在于所关注生成模型的多输入多输出（MIMO）特性。具体而言，在输入端，模型通常同时接收语音信号和角色引导（例如身份和情感），这不仅对学习能力构成挑战，也阻碍了后续对不同引导条件的适配；在输出端，整体人体运动主要由面部表情和身体动作组成，二者本质上相互关联，但在当前数据驱动的生成过程中难以协调。针对上述挑战，我们为两端分别提出了定制化设计。对于输入端，我们提出在固定身份和中性情感的数据上进行预训练，并将可定制条件（身份和情感）的引入推迟到微调阶段，同时通过我们新颖的 X-Adapter 实现参数高效微调来提升效果。对于输出端，我们提出了一种简单而有效的 Transformer 设计 DU-Trans，它首先分为两个分支分别学习面部表情和身体动作的个体特征，然后将二者联合起来学习双向联合分布，并直接预测组合系数。在 BEAT2 和 SHOW 数据集上的评估表明，Combo 不仅能高效生成高质量运动，还能高效迁移身份和情感。项目网站：\href{https://xc-csc101.github.io/combo/}{Combo}。

英文摘要：

In this paper, we propose a novel framework, Combo, for harmonious co-speech holistic 3D human motion generation and efficient customizable adaption. In particular, we identify that one fundamental challenge as the multiple-input-multiple-output (MIMO) nature of the generative model of interest. More concretely, on the input end, the model typically consumes both speech signals and character guidance (e.g., identity and emotion), which not only poses challenge on learning capacity but also hinders further adaptation to varying guidance; on the output end, holistic human motions mainly consist of facial expressions and body movements, which are inherently correlated but non-trivial to coordinate in current data-driven generation process. In response to the above challenge, we propose tailored designs to both ends. For the former, we propose to pre-train on data regarding a fixed identity with neutral emotion, and defer the incorporation of customizable conditions (identity and emotion) to fine-tuning stage, which is boosted by our novel X-Adapter for parameter-efficient fine-tuning. For the latter, we propose a simple yet effective transformer design, DU-Trans, which first divides into two branches to learn individual features of face expression and body movements, and then unites those to learn a joint bi-directional distribution and directly predicts combined coefficients. Evaluated on BEAT2 and SHOW datasets, Combo is highly effective in generating high-quality motions but also efficient in transferring identity and emotion. Project website: \href{https://xc-csc101.github.io/combo/}{Combo}.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。