DanceEditor：面向基于开放词汇描述的迭代可编辑音乐驱动舞蹈生成
DanceEditor: Towards Iterative Editable Music-driven Dance Generation with Open-Vocabulary Descriptions
AI总结：
针对现有音乐驱动舞蹈生成方法缺乏编辑功能的问题，构建DanceRemix多轮可编辑数据集，提出DanceEditor框架，通过预测-编辑范式整合音乐与文本条件，实现开放词汇描述下的迭代可编辑舞蹈生成并优于现有模型。
摘要：
从音乐信号生成连贯多样的人类舞蹈在虚拟化身动画领域取得了巨大进展。现有方法支持直接舞蹈合成，但未认识到在实际编舞场景中让用户编辑舞蹈动作更实用。此外，缺乏包含迭代编辑的高质量舞蹈数据集也限制了该挑战的解决。为此，我们首先构建DanceRemix数据集，这是一个大规模多轮可编辑舞蹈数据集，包含2530多万个舞蹈帧和84500对提示对。同时，我们提出新框架DanceEditor，基于预测-编辑范式整合多模态条件：初始预测阶段直接从匹配音乐建模舞蹈以提升结果权威性；后续迭代编辑阶段通过专门设计的跨模态编辑模块（CEM）整合文本描述作为条件信息，CEM自适应融合初始预测、音乐和文本提示作为时序运动线索引导合成序列，使结果既符合音乐节奏又保留与文本描述的细粒度语义对齐。大量实验表明，我们的方法在新收集的DanceRemix数据集上优于现有最先进模型。代码可在https://lzvsdy.github.io/DanceEditor/获取。
英文摘要：
Generating coherent and diverse human dances from music signals has gained tremendous progress in animating virtual avatars. While existing methods support direct dance synthesis, they fail to recognize that enabling users to edit dance movements is far more practical in real-world choreography scenarios. Moreover, the lack of high-quality dance datasets incorporating iterative editing also limits addressing this challenge. To achieve this goal, we first construct DanceRemix, a large-scale multi-turn editable dance dataset comprising the prompt featuring over 25.3M dance frames and 84.5K pairs. In addition, we propose a novel framework for iterative and editable dance generation coherently aligned with given music signals, namely DanceEditor. Considering the dance motion should be both musical rhythmic and enable iterative editing by user descriptions, our framework is built upon a prediction-then-editing paradigm unifying multi-modal conditions. At the initial prediction stage, our framework improves the authority of generated results by directly modeling dance movements from tailored, aligned music. Moreover, at the subsequent iterative editing stages, we incorporate text descriptions as conditioning information to draw the editable results through a specifically designed Cross-modality Editing Module (CEM). Specifically, CEM adaptively integrates the initial prediction with music and text prompts as temporal motion cues to guide the synthesized sequences. Thereby, the results display music harmonics while preserving fine-grained semantic alignment with text descriptions. Extensive experiments demonstrate that our method outperforms the state-of-the-art models on our newly collected DanceRemix dataset. Code is available at https://lzvsdy.github.io/DanceEditor/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
