增强视听视频字幕的时间与跨模态对齐
Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning
AI总结：
提出TCA-Captioner框架，通过观察-检查-纠正迭代策略和密集交互数据集，解决视听字幕中的模态分离与时间不一致问题，实现精准的视听绑定与因果动态建模。
摘要：
虽然多模态大语言模型（MLLMs）推动了视频理解的发展，但在视听视频字幕中实现精确的时间与跨模态对齐仍然是一个严峻的挑战。大多数现有方法存在模态分离和时间不一致的问题，无法准确地将听觉事件绑定到视觉实体，也无法捕捉复杂的因果动态。为了解决这些缺陷，我们提出了TCA-Captioner，这是一个专门为增强视听视频字幕的时间与跨模态对齐而设计的框架。我们首先引入了观察-检查-纠正（OCC）框架，这是一种迭代细化策略，用于生成高保真、精细标注的训练数据。利用精心策划的高密度人类交互数据集，TCA-Captioner被优化以建模复杂的视听交互。此外，我们提出了TCA-Bench，一个利用解耦评估协议来隔离和量化模型在视听绑定和时间关系推理方面能力的诊断基准。大量实验表明，TCA-Captioner为时间一致且同步的视听叙事设立了新标准。
英文摘要：
While Multimodal Large Language Models (MLLMs) have advanced video understanding, achieving precise temporal and cross-modal alignment in audiovisual video captioning remains a formidable challenge. Most existing approaches suffer from modality detachment and temporal incoherence, failing to accurately bind auditory events to visual entities or capture complex causal dynamics. To address these deficiencies, we propose TCA-Captioner, a framework specifically engineered to enhance Temporal and Cross-Modal Alignment for audiovisual video captioning. We first introduce the Observer-Checker-Corrector (OCC) framework, an iterative refinement strategy that generates high-fidelity, meticulously grounded training data. Leveraging a curated high-density human interaction dataset, TCA-Captioner is optimized to model sophisticated audiovisual interactions. Furthermore, we present TCA-Bench, a diagnostic benchmark utilizing a Decoupled Evaluation Protocol to isolate and quantify model proficiency in audiovisual binding and temporal relational reasoning. Extensive experiments demonstrate that TCA-Captioner sets a new standard for temporally-coherent and synchronized audiovisual narratives.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
