NES视频-音乐数据库：一个与游戏玩法视频配对的符号化电子游戏音乐数据集
The NES Video-Music Database: A Dataset of Symbolic Video Game Music Paired with Gameplay Videos
AI总结：
本文构建NES-VMDB数据集，将98,940个NES游戏视频片段与MIDI配乐配对，并提出条件式CMT基线，验证玩法视频可改善音乐结构与类型关联学习。
摘要：
神经模型是音乐生成中最受欢迎的方法之一，但目前尚缺乏专门用于直接从游戏数据中学习音乐的标准大型数据集。为填补这一研究空白，我们提出了一个名为NES-VMDB的新型数据集，其中包含来自389款NES游戏的98,940个游戏玩法视频，每个视频都与其符号格式（MIDI）的原声配乐配对。NES-VMDB建立在Nintendo Entertainment System Music Database（NES-MDB）之上，涵盖来自397款NES游戏的5,278首音乐作品。我们的方法包括为原始数据集中的389款游戏收集长时游玩视频，将其切片为15秒长的片段，并从每个片段中提取音频。随后，我们应用一种音频指纹识别算法（类似于Shazam）来自动识别NES-MDB数据集中对应的乐曲。此外，我们提出了一种基于Controllable Music Transformer的基线方法，用于根据游戏玩法片段生成NES音乐。我们使用客观指标评估了该方法，结果表明，与无条件版本相比，条件式CMT提升了音乐结构质量。此外，我们使用神经分类器预测所生成乐曲的游戏类型。结果显示，CMT生成器能够学习游戏玩法视频与游戏类型之间的关联，但要达到人类水平的性能仍需开展进一步研究。
英文摘要：
Neural models are one of the most popular approaches for music generation, yet there aren't standard large datasets tailored for learning music directly from game data. To address this research gap, we introduce a novel dataset named NES-VMDB, containing 98,940 gameplay videos from 389 NES games, each paired with its original soundtrack in symbolic format (MIDI). NES-VMDB is built upon the Nintendo Entertainment System Music Database (NES-MDB), encompassing 5,278 music pieces from 397 NES games. Our approach involves collecting long-play videos for 389 games of the original dataset, slicing them into 15-second-long clips, and extracting the audio from each clip. Subsequently, we apply an audio fingerprinting algorithm (similar to Shazam) to automatically identify the corresponding piece in the NES-MDB dataset. Additionally, we introduce a baseline method based on the Controllable Music Transformer to generate NES music conditioned on gameplay clips. We evaluated this approach with objective metrics, and the results showed that the conditional CMT improves musical structural quality when compared to its unconditional counterpart. Moreover, we used a neural classifier to predict the game genre of the generated pieces. Results showed that the CMT generator can learn correlations between gameplay videos and game genres, but further research has to be conducted to achieve human-level performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
