从对比到共同性：用于增强多模态大语言模型中音频-文本跨模态理解的音频共同性描述
From Contrast to Commonality: Audio Commonality Captioning for Enhanced Audio-Text Cross-modal Understanding in Multimodal LLMs
AI总结：
本文提出音频共同性描述方法，旨在增强多模态大语言模型中音频与文本的跨模态理解，通过捕捉音频片段间的共享语义以改善模型的泛化能力与任务特定性能。
摘要：
音频描述（AC）在多模态大语言模型（MLLMs）的预训练和微调过程中扮演着关键角色。为了加强这种对齐，最近的研究提出了音频差异描述（ADC），它通过多个音频输入并鼓励模型描述它们的差异，从而促进细粒度的辨别。然而，尽管其有效性，ADC在输入音频（通常富含多样事件）和简短的、以差异为中心的短描述之间引入了语义差距。这种偏离AC式任务导致与预训练目标的不匹配，从而导致灾难性遗忘。为了解决这个问题，我们提出了音频共同性描述（ACC），这是一种同样具有挑战性但更温和的替代方法，它引导模型捕捉音频片段之间的共享语义，而不是详细的差异。实验表明，ACC不仅在描述基准上提高了音频-文本理解，还更好地保持了在多样语音和音乐任务中的通用能力，证实了其在增强更稳健的跨模态理解以及在MLLMs中实现通用性和任务特定性能之间取得更好平衡的能力。
英文摘要：
Audio Captioning (AC) plays a pivotal role in enhancing audio-text cross-modal understanding during the pretraining and finetuning of Multimodal LLMs (MLLMs). To strengthen this alignment, recent works propose Audio Difference Captioning (ADC), which takes multiple audio inputs and encourages the model to describe their differences, thereby promoting fine-grained discrimination. However, despite its effectiveness, ADC introduces a semantic gap between input audios-often rich in diverse events-and the brief, difference-focused short caption. This deviation from AC-style task causes a mismatch with the pretraining objective, leading to catastrophic forgetting. To address this, we propose Audio Commonality Captioning (ACC), a comparably challenging but gentler alternative that guides the model to capture shared semantics across audio clips rather than detailed differences. Experiments show that ACC not only improves audio-text understanding on captioning benchmarks but also better preserves general capabilities across diverse speech and music tasks, confirming its ability to enable more robust cross-modal understanding and achieve a better balance between generalization and task-specific performance in MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
