LAVCap：基于最优传输的 LLM 音视频描述生成
LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport
AI总结：
本文提出基于 LLM 的音视频描述框架 LAVCap，通过最优传输对齐损失与注意力模块增强跨模态融合，在 AudioCaps 上超越现有最先进方法。
摘要：
自动音频描述是一项为音频内容生成文本描述的任务，近期研究已探索利用视觉信息来提升描述质量。然而，现有方法通常无法有效融合音频与视觉数据，从而遗漏各模态中的重要语义线索。为解决该问题，我们提出 LAVCap，这是一个基于大型语言模型（LLM）的音视频描述生成框架，可有效整合视觉信息与音频，以提升音频描述性能。LAVCap 采用基于最优传输的对齐损失来弥合音频特征与视觉特征之间的模态差距，从而实现更有效的语义提取。此外，我们提出一个最优传输注意力模块，利用最优传输分配图增强音视频融合。结合最优训练策略，实验结果表明我们框架的每个组件都是有效的。LAVCap 在 AudioCaps 数据集上优于现有最先进方法，且无需依赖大型数据集或后处理。代码可在 https://github.com/NAVER-INTEL-Co-Lab/gaudi-lavcap 获取。
英文摘要：
Automated audio captioning is a task that generates textual descriptions for audio content, and recent studies have explored using visual information to enhance captioning quality. However, current methods often fail to effectively fuse audio and visual data, missing important semantic cues from each modality. To address this, we introduce LAVCap, a large language model (LLM)-based audio-visual captioning framework that effectively integrates visual information with audio to improve audio captioning performance. LAVCap employs an optimal transport-based alignment loss to bridge the modality gap between audio and visual features, enabling more effective semantic extraction. Additionally, we propose an optimal transport attention module that enhances audio-visual fusion using an optimal transport assignment map. Combined with the optimal training strategy, experimental results demonstrate that each component of our framework is effective. LAVCap outperforms existing state-of-the-art methods on the AudioCaps dataset, without relying on large datasets or post-processing. Code is available at https://github.com/NAVER-INTEL-Co-Lab/gaudi-lavcap.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
