SKY-Piano：多模态钢琴演奏数据集
SKY-Piano: A Multimodal Piano Performance Dataset
- Seoul National University（首尔大学）
- KAIST（韩国科学技术院）
- Yamaha（雅马哈）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
该研究发布SKY-Piano多模态钢琴演奏数据集，含多类数据及标注工具，通过微调实验验证其用于MIDI到动作生成的可用性。
AI中文摘要：
音乐信息检索领域对钢琴演奏的研究，越来越多地涉及音频和MIDI之外的多种数据模态与标注。我们提出SKY-Piano，这是一个多模态钢琴演奏数据集，包含7位专业钢琴家与12位业余钢琴家演奏的11小时表演记录，涵盖动作、多视角视频、音频、MIDI数据，以及MusicXML乐谱。表演曲目结合演奏技巧、难度、演奏者专业度，从共享核心曲目中选取。动作数据包含手部与身体动作，以两种形式发布：标记形式（标记因标记物遮挡丢失的样本为不可靠）和插补形式（重建这些缺失部分），同时包含Visual3D身体段运动学及其他时间同步模态。为便于快速浏览不同数据模态，我们提供交互式网络浏览器。此外，我们开发了指法标注模型与工具，用于从MIDI和动作数据生成伪指法标注。最后，我们通过微调实验展示MIDI到动作的生成，作为该数据集的一个应用案例。
英文摘要：
Music information retrieval research on piano performance increasingly involves diverse modalities of data and annotations beyond audio and MIDI. We present SKY-Piano, a multimodal piano performance dataset that includes 11 hours of performance recordings of motion, multi-view video, audio, MIDI from 7 professional and 12 amateur pianists along with MusicXML scores. The performance pieces were selected considering playing technique, difficulty, and performer expertise on a shared core repertoire. The motion data include both hand and body motion, released in both flagged form, where samples lost to marker occlusion are marked as unreliable, and imputed form, where those gaps are reconstructed, together with Visual3D body-segment kinematics and other time-synchronized modalities. To easily browse different modalities of data at a glance, we provide an interactive web browser. In addition, we developed a fingering annotation model and tool for deriving pseudo fingering annotations from the MIDI and motion data. Lastly, we present MIDI-to-motion generation through a fine-tuning experiment as a use case of the dataset.
