FürElise: 捕捉并物理合成钢琴演奏的手部动作
FürElise: Capturing and Physically Synthesizing Hand Motions of Piano Performance
AI总结：
本研究构建了首个大规模顶尖钢琴家3D手部演奏动作数据集，提出结合扩散模型、模仿与强化学习的物理合成方法，可生成自然灵巧的手部动作并泛化到训练外的乐谱。
摘要：
钢琴演奏需要灵活、精准且协调的手部控制，这对动作灵巧度提出了极高要求。能够精准复现钢琴演奏的精细手部动作模型，在角色动画、具身AI、生物力学以及VR/AR领域有着广泛的应用前景。本文构建了首个大规模钢琴演奏手部动作数据集，包含15位顶尖钢琴家演奏153首古典乐曲的约10小时3D手部动作与音频数据。为捕捉自然的演奏状态，我们设计了无标记捕捉方案，采用前沿姿态估计模型从多视角视频中重建动作，并借助特制Yamaha Disklavier钢琴传感器采集的高分辨率MIDI按键数据，通过逆运动学进一步优化动作数据。依托该数据集，我们开发了一套流水线，可为数据集外的乐谱合成符合物理规律的手部动作。该方法结合模仿学习与强化学习，得到基于物理的双手控制策略，实现手部与钢琴琴键的交互。为解决大规模动作数据集的采样效率问题，我们使用扩散模型生成自然参考动作，提供高层轨迹与指法（手指顺序与位置）信息，但仅靠生成的参考动作无法满足钢琴演奏建模的精度要求，因此我们进一步利用音乐相似度从采集的数据集中检索相似动作进行数据增强，以提升RL策略的精度。通过所提方法，我们的模型能生成自然、灵巧的动作，且可泛化至训练数据集外的乐曲。
英文摘要：
Piano playing requires agile, precise, and coordinated hand control that stretches the limits of dexterity. Hand motion models with the sophistication to accurately recreate piano playing have a wide range of applications in character animation, embodied AI, biomechanics, and VR/AR. In this paper, we construct a first-of-its-kind large-scale dataset that contains approximately 10 hours of 3D hand motion and audio from 15 elite-level pianists playing 153 pieces of classical music. To capture natural performances, we designed a markerless setup in which motions are reconstructed from multi-view videos using state-of-the-art pose estimation models. The motion data is further refined via inverse kinematics using the high-resolution MIDI key-pressing data obtained from sensors in a specialized Yamaha Disklavier piano. Leveraging the collected dataset, we developed a pipeline that can synthesize physically-plausible hand motions for musical scores outside of the dataset. Our approach employs a combination of imitation learning and reinforcement learning to obtain policies for physics-based bimanual control involving the interaction between hands and piano keys. To solve the sampling efficiency problem with the large motion dataset, we use a diffusion model to generate natural reference motions, which provide high-level trajectory and fingering (finger order and placement) information. However, the generated reference motion alone does not provide sufficient accuracy for piano performance modeling. We then further augmented the data by using musical similarity to retrieve similar motions from the captured dataset to boost the precision of the RL policy. With the proposed method, our model generates natural, dexterous motions that generalize to music from outside the training dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
