ADIFF：使用自然语言解释音频差异
ADIFF: Explaining audio difference using natural language
AI总结：
本文首次系统研究音频差异解释任务，提出ADIFF模型，通过交叉投影、位置字幕和三步训练生成多层次自然语言解释，并在新基准上显著超越基线方法。
摘要：
理解和解释音频录音之间的差异对于音频取证、质量评估和音频生成等领域至关重要。这涉及识别和描述音频事件、声学场景、信号特征及其对听众的情感影响。本文首次全面研究了音频差异解释任务，并为该任务提出了基准和基线。首先，我们基于AudioCaps和Clotho音频字幕数据集，提出了两个新的音频差异解释数据集。利用大型语言模型（LLMs），我们生成三个层次的差异解释：（1）音频事件和对象的简洁描述，（2）关于音频事件、声学场景和信号属性的简短句子，以及（3）包含语义和听众情感的综合解释。对于基线，我们使用前缀调优，将两个音频文件的音频嵌入用于提示一个冻结的语言模型。我们的实证分析和消融研究表明，朴素基线难以区分感知上相似的声音，并且难以生成详细的第三层解释。为了解决这些局限性，我们提出了ADIFF，它引入了交叉投影模块、位置字幕和三步训练过程，以增强模型生成详细解释的能力。我们使用客观指标和人工评估来评估我们的模型，并表明我们的模型增强在性能上显著优于朴素基线和最先进的音频-语言模型（ALM）Qwen Audio。最后，我们进行了多项消融研究，以研究交叉投影、语言模型参数、位置字幕、第三阶段微调的影响，并呈现了我们的发现。我们的基准、发现和强大的基线为音频差异的细致且类人的解释铺平了道路。
英文摘要：
Understanding and explaining differences between audio recordings is crucial for fields like audio forensics, quality assessment, and audio generation. This involves identifying and describing audio events, acoustic scenes, signal characteristics, and their emotional impact on listeners. This paper stands out as the first work to comprehensively study the task of explaining audio differences and then propose benchmark, baselines for the task. First, we present two new datasets for audio difference explanation derived from the AudioCaps and Clotho audio captioning datasets. Using Large Language Models (LLMs), we generate three levels of difference explanations: (1) concise descriptions of audio events and objects, (2) brief sentences about audio events, acoustic scenes, and signal properties, and (3) comprehensive explanations that include semantics and listener emotions. For the baseline, we use prefix tuning where audio embeddings from two audio files are used to prompt a frozen language model. Our empirical analysis and ablation studies reveal that the naive baseline struggles to distinguish perceptually similar sounds and generate detailed tier 3 explanations. To address these limitations, we propose ADIFF, which introduces a cross-projection module, position captioning, and a three-step training process to enhance the model's ability to produce detailed explanations. We evaluate our model using objective metrics and human evaluation and show our model enhancements lead to significant improvements in performance over naive baseline and SoTA Audio-Language Model (ALM) Qwen Audio. Lastly, we conduct multiple ablation studies to study the effects of cross-projection, language model parameters, position captioning, third stage fine-tuning, and present our findings. Our benchmarks, findings, and strong baseline pave the way for nuanced and human-like explanations of audio differences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
