EmotionCaps：通过情感增强的数据生成提升音频字幕生成
EmotionCaps: Enhancing Audio Captioning Through Emotion-Augmented Data Generation
AI总结：
本研究提出EmotionCaps数据集，通过ChatGPT结合声景情感信息生成约12万条情感增强的音频字幕数据，实验证明该数据能提升字幕模型性能并匹配音频情感基调。
摘要：
音频语言建模的最新进展（如自动音频字幕生成）得益于在大型语言模型辅助生成的合成数据上进行训练。然而，针对环境声音字幕生成的此类方法主要关注音频事件标签，并未探索利用录音中可能存在的情感信息。在本研究中，我们通过以估计的声景情感这一额外声学信息指示ChatGPT，探索生成情感增强合成音频字幕数据的益处。为此，我们引入了EmotionCaps，这是一个音频字幕数据集，包含约120,000个音频片段，配有融合了声景情感识别（SER）信息的合成描述。我们假设这些额外信息将产生更高质量的、与音频录音情感基调相匹配的字幕，进而提升基于此数据训练的字幕模型性能。我们通过客观和主观评估测试了该假设，将使用EmotionCaps数据集训练的模型与多个基线模型进行了比较。研究结果挑战了当前的字幕生成方法，并为开发和评估字幕模型提出了新方向。
英文摘要：
Recent progress in audio-language modeling, such as automated audio captioning, has benefited from training on synthetic data generated with the aid of large-language models. However, such approaches for environmental sound captioning have primarily focused on audio event tags and have not explored leveraging emotional information that may be present in recordings. In this work, we explore the benefit of generating emotion-augmented synthetic audio caption data by instructing ChatGPT with additional acoustic information in the form of estimated soundscape emotion. To do so, we introduce EmotionCaps, an audio captioning dataset comprised of approximately 120,000 audio clips with paired synthetic descriptions enriched with soundscape emotion recognition (SER) information. We hypothesize that this additional information will result in higher-quality captions that match the emotional tone of the audio recording, which will, in turn, improve the performance of captioning models trained with this data. We test this hypothesis through both objective and subjective evaluation, comparing models trained with the EmotionCaps dataset to multiple baseline models. Our findings challenge current approaches to captioning and suggest new directions for developing and assessing captioning models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
