M2M-Gen：一种使用大语言模型为日本漫画自动生成背景音乐的多模态框架
M2M-Gen: A Multimodal Framework for Automated Background Music Generation in Japanese Manga Using Large Language Models
AI总结：
本文提出M2M Gen多模态框架，通过对话检测和情感分类提取场景信息，并利用GPT4o生成音乐指令指导文本到音乐模型，为日本漫画自动生成高质量且契合叙事的背景音乐。
摘要：
本文介绍了M2M Gen，这是一个为日本漫画量身定制生成背景音乐的多模态框架。该任务的关键挑战在于缺乏可用的数据集或基线。为了应对这些挑战，我们提出了一种自动音乐生成流水线，能够为输入的漫画书生成背景音乐。首先，我们利用漫画中的对话来检测场景边界，并使用场景内角色的面部进行情感分类。然后，我们使用GPT4o将这些低级场景信息转化为高级音乐指令。在场景信息和音乐指令的条件下，另一个GPT 4o实例生成页面级音乐字幕，以指导文本到音乐模型。这产生了与漫画不断演变的叙事相一致的音乐。通过广泛的主观评估，证实了M2M Gen的有效性，展示了其能够生成更高质量、更相关且更一致的音乐，与我们的基线相比，能更好地补充特定场景。
英文摘要：
This paper introduces M2M Gen, a multi modal framework for generating background music tailored to Japanese manga. The key challenges in this task are the lack of an available dataset or a baseline. To address these challenges, we propose an automated music generation pipeline that produces background music for an input manga book. Initially, we use the dialogues in a manga to detect scene boundaries and perform emotion classification using the characters faces within a scene. Then, we use GPT4o to translate this low level scene information into a high level music directive. Conditioned on the scene information and the music directive, another instance of GPT 4o generates page level music captions to guide a text to music model. This produces music that is aligned with the mangas evolving narrative. The effectiveness of M2M Gen is confirmed through extensive subjective evaluations, showcasing its capability to generate higher quality, more relevant and consistent music that complements specific scenes when compared to our baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
