零努力图像到音乐生成：一种可解释的基于RAG的视觉语言模型方法
Zero-Effort Image-to-Music Generation: An Interpretable RAG-based VLM Approach
AI总结：
本文提出一种基于RAG的视觉语言模型方法，实现图像到音乐生成，通过ABC记谱法连接文本与音乐模态，并利用多模态检索增强生成和自反思技术，提供高可解释性且低计算成本的音乐生成方案。
摘要：
近年来，图像到音乐（I2M）生成引起了广泛关注，潜在应用于游戏、广告和多模态艺术创作领域。然而，由于I2M任务的模糊性和主观性，大多数端到端方法缺乏可解释性，使用户困惑于生成结果。即使基于情感映射的方法也存在争议，因为情感仅代表艺术的一个方面。此外，大多数基于学习的方法需要大量计算资源和大数据集进行训练，限制了普通用户对它们的访问性。为了解决这些挑战，我们提出了首个基于视觉语言模型（VLM）的I2M框架，提供高可解释性和低计算成本。具体来说，我们利用ABC记谱法连接文本和音乐模态，使VLM能够使用自然语言生成音乐。然后，我们应用多模态检索增强生成（RAG）和自反思技术，使VLM能够在不外部训练的情况下生成高质量音乐。此外，我们利用生成的文本动机和VLM的注意力图，为生成结果在文本和图像模态中提供解释。为了验证我们的方法，我们进行了人类研究和机器评估，其中我们的方法在音乐质量和音乐-图像一致性方面优于其他方法，表明了有前途的结果。我们的代码可在https://github.com/RS2002/Image2Music上获得。
英文摘要：
Recently, Image-to-Music (I2M) generation has garnered significant attention, with potential applications in fields such as gaming, advertising, and multi-modal art creation. However, due to the ambiguous and subjective nature of I2M tasks, most end-to-end methods lack interpretability, leaving users puzzled about the generation results. Even methods based on emotion mapping face controversy, as emotion represents only a singular aspect of art. Additionally, most learning-based methods require substantial computational resources and large datasets for training, hindering accessibility for common users. To address these challenges, we propose the first Vision Language Model (VLM)-based I2M framework that offers high interpretability and low computational cost. Specifically, we utilize ABC notation to bridge the text and music modalities, enabling the VLM to generate music using natural language. We then apply multi-modal Retrieval-Augmented Generation (RAG) and self-refinement techniques to allow the VLM to produce high-quality music without external training. Furthermore, we leverage the generated motivations in text and the attention maps from the VLM to provide explanations for the generated results in both text and image modalities. To validate our method, we conduct both human studies and machine evaluations, where our method outperforms others in terms of music quality and music-image consistency, indicating promising results. Our code is available at https://github.com/RS2002/Image2Music .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
