Art2Music: 通过多模态情感对齐生成艺术图像的音乐
Art2Music: Generating Music for Art Images with Multi-modal Feeling Alignment
AI总结：
Art2Music通过多模态情感对齐生成艺术图像的音乐，利用轻量级跨模态框架提升音乐生成的感知自然性和频谱保真度。
摘要：
随着AI生成内容（AIGC）的兴起，从多模态输入中生成感知自然且情感对齐的音乐已成为核心挑战。现有方法往往依赖于显式情绪标签，需要昂贵的标注，凸显了需要更灵活的情感对齐方法的必要性。为了支持多模态音乐生成，我们构建了ArtiCaps，一个通过语义匹配ArtEmis和MusicCaps的描述来创建的伪情感对齐图像-音乐-文本数据集。我们进一步提出了Art2Music，一个轻量级的跨模态框架，能够从艺术图像和用户评论中合成音乐。在第一阶段，图像和文本通过OpenCLIP编码，并通过门控残差模块融合；融合的表示通过双向LSTM解码为梅尔频谱图，并通过频率加权L1损失增强高频保真度。在第二阶段，经过微调的HiFi-GAN vocoder重建高质量的音频波形。在ArtiCaps上的实验显示，在梅尔-cepstral失真、Fréchet音频距离、对数频谱距离和余弦相似度方面均有明显改进。一项小规模基于LLM的评分研究进一步验证了跨模态情感对齐的一致性，并提供了跨模态匹配和不匹配的可解释解释。这些结果表明在感知自然性、频谱保真度和语义一致性方面有所提高。Art2Music在仅有50k训练样本的情况下仍能保持稳健的性能，为交互艺术、个性化声音景观和数字艺术展览中的情感对齐创意音频生成提供了可扩展的解决方案。
英文摘要：
With the rise of AI-generated content (AIGC), generating perceptually natural and feeling-aligned music from multimodal inputs has become a central challenge. Existing approaches often rely on explicit emotion labels that require costly annotation, underscoring the need for more flexible feeling-aligned methods. To support multimodal music generation, we construct ArtiCaps, a pseudo feeling-aligned image-music-text dataset created by semantically matching descriptions from ArtEmis and MusicCaps. We further propose Art2Music, a lightweight cross-modal framework that synthesizes music from artistic images and user comments. In the first stage, images and text are encoded with OpenCLIP and fused using a gated residual module; the fused representation is decoded by a bidirectional LSTM into Mel-spectrograms with a frequency-weighted L1 loss to enhance high-frequency fidelity. In the second stage, a fine-tuned HiFi-GAN vocoder reconstructs high-quality audio waveforms. Experiments on ArtiCaps show clear improvements in Mel-Cepstral Distortion, Frechet Audio Distance, Log-Spectral Distance, and cosine similarity. A small LLM-based rating study further verifies consistent cross-modal feeling alignment and offers interpretable explanations of matches and mismatches across modalities. These results demonstrate improved perceptual naturalness, spectral fidelity, and semantic consistency. Art2Music also maintains robust performance with only 50k training samples, providing a scalable solution for feeling-aligned creative audio generation in interactive art, personalized soundscapes, and digital art exhibitions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
