Listening without Looking：视听字幕生成中的模态偏差
Listening without Looking: Modality Bias in Audio-Visual Captioning
AI总结：
该研究针对视听字幕生成模型的模态互补性与鲁棒性问题，通过对LAVCap开展模态鲁棒性测试发现其存在音频偏向，还构建了AudioVisualCaps数据集，实验证明在该数据集上训练的LAVCap模态偏差更低。
摘要：
视听字幕生成旨在通过联合建模声音与视觉，生成完整的场景描述。尽管近期方法通过复杂的模态融合提升了性能，但目前尚不清楚在现有视听字幕生成模型中，两种模态的互补程度如何，以及当其中一种模态质量下降时，这些模型的鲁棒性如何。我们通过对当前最先进的视听字幕生成模型LAVCap开展系统性的模态鲁棒性测试来解答这些问题，测试中我们选择性地抑制或破坏音频或视觉流，以量化模型的敏感度与模态互补性。分析显示，LAVCap存在明显的音频流偏向。为评估视听字幕生成模型对两种模态的使用均衡度，我们为AudioCaps补充了可同时描述音频与视觉流的文本标注，构建出AudioVisualCaps数据集。实验中，我们报告了LAVCap在AudioVisualCaps上的基线结果，同时在AudioVisualCaps上开展模态鲁棒性测试评估该模型，结果表明，在AudioVisualCaps上训练的LAVCap，其模态偏差比在AudioCaps上训练时更小。
英文摘要：
Audio-visual captioning aims to generate holistic scene descriptions by jointly modeling sound and vision. While recent methods have improved performance through sophisticated modality fusion, it remains unclear to what extent the two modalities are complementary in current audio-visual captioning models and how robust these models are when one modality is degraded. We address these questions by conducting systematic modality robustness tests on LAVCap, a state-of-the-art audio-visual captioning model, in which we selectively suppress or corrupt the audio or visual streams to quantify sensitivity and complementarity. The analysis reveals a pronounced bias toward the audio stream in LAVCap. To evaluate how balanced audio-visual captioning models are in their use of both modalities, we augment AudioCaps with textual annotations that jointly describe the audio and visual streams, yielding the AudioVisualCaps dataset. In our experiments, we report LAVCap baseline results on AudioVisualCaps. We also evaluate the model under modality robustness tests on AudioVisualCaps and the results indicate that LAVCap trained on AudioVisualCaps exhibits less modality bias than when trained on AudioCaps.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
