音频字幕生成研究：解决词汇不平衡问题并评估以语言为中心的性能指标的适用性
Investigations in Audio Captioning: Addressing Vocabulary Imbalance and Evaluating Suitability of Language-Centric Performance Metrics
AI总结：
针对自动音频字幕生成面临的数据稀缺、词汇不平衡、评估指标不适配三大挑战，提出两种数据与词汇增强方法，验证领域内预训练的必要性，并指出五种图像字幕指标在音频领域的局限。
摘要：
对周围声音中的有效信息进行分析、处理与提取是音频分析这一广泛领域的研究主题。音频字幕生成是音频分析领域的新兴方向，属于跨模态翻译任务，核心是为音频流中出现的声音事件生成自然语言描述。本研究明确了自动音频字幕生成的三大主要挑战并加以改进：一是数据稀缺，二是音频字幕词汇存在不平衡或局限性，三是缺乏能同时兼顾听觉特征与语义特征的合适性能评估指标。研究发现，常用的损失函数会在模型训练过程中导致不合理的词汇不平衡问题。为此，我们提出两种音频字幕生成增强方法，可扩充训练数据集并丰富词汇量。此外，我们通过探究此前在不同音频任务上预训练的音频编码器的适用性，进一步强调了领域内预训练的必要性。最后，我们系统研究了从图像字幕生成领域借鉴的五种性能指标，并指出了它们在音频领域的局限性。
英文摘要：
The analysis, processing, and extraction of meaningful information from sounds all around us is the subject of the broader area of audio analytics. Audio captioning is a recent addition to the domain of audio analytics, a cross-modal translation task that focuses on generating natural descriptions from sound events occurring in an audio stream. In this work, we identify and improve on three main challenges in automated audio captioning: i) data scarcity, ii) imbalance or limitations in the audio captions vocabulary, and iii) the proper performance evaluation metric that can best capture both auditory and semantic characteristics. We find that generally adopted loss functions can result in an unfair vocabulary imbalance during model training. We propose two audio captioning augmentation methods that enrich the training dataset and the vocabulary size. We further underline the need for in-domain pretraining by exploring the suitability of audio encoders that were previously trained on different audio tasks. Finally, we systematically explore five performance metrics borrowed from the image captioning domain and highlight their limitations for the audio domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
