基于Skeleton Mamba的EgoMusic驱动人体舞蹈动作估计
EgoMusic-driven Human Dance Motion Estimation with Skeleton Mamba
AI总结：
本文提出一种从自我中心视频和音乐联合预测人体舞蹈动作的新方法，构建了大规模数据集EgoAIST++，并开发了基于扩散模型和Skeleton Mamba的EgoMusic动作网络，实验证明其优于现有方法且能泛化至真实数据。
摘要：
估计人体舞蹈动作是一项具有各种工业应用的挑战性任务。最近，许多研究致力于使用自我中心视频或音乐作为输入来预测人体舞蹈动作。然而，联合从自我中心视频和音乐中估计人体动作的任务在很大程度上仍未被探索。在本文中，我们旨在开发一种新方法，从自我中心视频和音乐中预测人体舞蹈动作。在实践中，自我中心视角通常会遮挡大部分身体，使得准确的全姿态估计具有挑战性。此外，结合音乐要求生成的头部和身体动作与视觉和音乐输入良好对齐。我们首先引入了EgoAIST++，这是一个新的大规模数据集，结合了自我中心视角和音乐，包含超过36小时的舞蹈动作。基于扩散模型和Mamba在序列建模上的成功，我们开发了一个EgoMusic动作网络，其核心是Skeleton Mamba，能够显式捕捉人体的骨架结构。我们证明了我们的方法在理论上是支持的。大量实验表明，我们的方法明显优于最先进的方法，并有效泛化到真实世界数据。
英文摘要：
Estimating human dance motion is a challenging task with various industrial applications. Recently, many efforts have focused on predicting human dance motion using either egocentric video or music as input. However, the task of jointly estimating human motion from both egocentric video and music remains largely unexplored. In this paper, we aim to develop a new method that predicts human dance motion from both egocentric video and music. In practice, the egocentric view often obscures much of the body, making accurate full-pose estimation challenging. Additionally, incorporating music requires the generated head and body movements to align well with both visual and musical inputs. We first introduce EgoAIST++, a new large-scale dataset that combines both egocentric views and music with more than 36 hours of dancing motion. Drawing on the success of diffusion models and Mamba on modeling sequences, we develop an EgoMusic Motion Network with a core Skeleton Mamba that explicitly captures the skeleton structure of the human body. We illustrate that our approach is theoretically supportive. Intensive experiments show that our method clearly outperforms state-of-the-art approaches and generalizes effectively to real-world data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
