Meerkat：面向空间与时间定位的音视频大语言模型
Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time
AI总结：
本文提出音视频大语言模型 Meerkat，通过最优传输模态对齐和交叉注意力模块实现细粒度空间与时间理解，并构建 AVFIT 数据集与 MeerkatBench 基准，在多个音视频定位与核查任务上取得领先性能。
摘要：
利用大语言模型在文本任务上的卓越能力，近期多模态大语言模型（MLLMs）的研究将其扩展到视觉和音频等其他模态。然而，这些方向的进展主要集中在仅需对音视频语义进行粗粒度理解的任务上。我们提出 Meerkat，一个具备对图像和音频在空间与时间上进行细粒度理解的音视频大语言模型。通过基于最优传输的新模态对齐模块和强制音视频一致性的交叉注意力模块，Meerkat 能够应对具有挑战性的任务，如音频指代图像定位、图像引导的音频时间定位以及音视频事实核查。此外，我们精心构建了一个大规模数据集 AVFIT，包含从开源数据集收集的 3M 条指令微调样本，并引入 MeerkatBench 统一了五个具有挑战性的音视频任务。我们在所有这些下游任务上取得了最先进的性能，相对提升最高达 37.12%。
英文摘要：
Leveraging Large Language Models' remarkable proficiency in text-based tasks, recent works on Multi-modal LLMs (MLLMs) extend them to other modalities like vision and audio. However, the progress in these directions has been mostly focused on tasks that only require a coarse-grained understanding of the audio-visual semantics. We present Meerkat, an audio-visual LLM equipped with a fine-grained understanding of image and audio both spatially and temporally. With a new modality alignment module based on optimal transport and a cross-attention module that enforces audio-visual consistency, Meerkat can tackle challenging tasks such as audio referred image grounding, image guided audio temporal localization, and audio-visual fact-checking. Moreover, we carefully curate a large dataset AVFIT that comprises 3M instruction tuning samples collected from open-source datasets, and introduce MeerkatBench that unifies five challenging audio-visual tasks. We achieve state-of-the-art performance on all these downstream tasks with a relative improvement of up to 37.12%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
