面向视频到钢琴音乐生成：基于表演链支持基准
Towards Video to Piano Music Generation with Chain-of-Perform Support Benchmarks
- AI Lab, Giant Network（AI实验室，巨人网络）
- University of Trento（特伦托大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
针对视频生成钢琴音乐缺乏专门基准的问题，提出CoP基准数据集，提供多模态标注、多功能评估框架及完全开源，以促进高质量生成研究。
AI中文摘要：
从视频生成高质量钢琴音频需要视觉线索与音乐输出之间的精确同步，确保语义和时间上的准确对齐。然而，现有的评估数据集并未完全捕捉钢琴音乐生成所需的复杂同步性。一个全面的基准对于两个主要原因至关重要：（1）现有指标未能反映视频到钢琴音乐交互的复杂性，（2）专门的基准数据集可以提供有价值的见解，以加速高质量钢琴音乐生成的进展。为解决这些挑战，我们引入了CoP基准数据集——一个完全开源的多模态基准，专门设计用于视频引导的钢琴音乐生成。所提出的表演链（CoP）基准提供了几个引人注目的特性：（1）详细的多模态标注，通过逐步的表演链指导实现视频内容与钢琴音频之间的精确语义和时间对齐；（2）一个多功能的评估框架，用于严格评估通用和专门的视频到钢琴生成任务；（3）数据集、标注和评估协议的完全开源。该数据集可在https://github.com/acappemin/Video-to-Audio-and-Piano公开获取，并配有持续更新的排行榜，以促进该领域的持续研究。
英文摘要：
Generating high-quality piano audio from video requires precise synchronization between visual cues and musical output, ensuring accurate semantic and temporal alignment.However, existing evaluation datasets do not fully capture the intricate synchronization required for piano music generation. A comprehensive benchmark is essential for two primary reasons: (1) existing metrics fail to reflect the complexity of video-to-piano music interactions, and (2) a dedicated benchmark dataset can provide valuable insights to accelerate progress in high-quality piano music generation. To address these challenges, we introduce the CoP Benchmark Dataset-a fully open-sourced, multimodal benchmark designed specifically for video-guided piano music generation. The proposed Chain-of-Perform (CoP) benchmark offers several compelling features: (1) detailed multimodal annotations, enabling precise semantic and temporal alignment between video content and piano audio via step-by-step Chain-of-Perform guidance; (2) a versatile evaluation framework for rigorous assessment of both general-purpose and specialized video-to-piano generation tasks; and (3) full open-sourcing of the dataset, annotations, and evaluation protocols. The dataset is publicly available at https://github.com/acappemin/Video-to-Audio-and-Piano, with a continuously updated leaderboard to promote ongoing research in this domain.
