R-AVST：通过细粒度时空推理增强视频大语言模型以应对复杂音频视觉场景
R-AVST: Empowering Video-LLMs with Fine-Grained Spatio-Temporal Reasoning in Complex Audio-Visual Scenarios
AI总结：
R-AVST通过细粒度时空推理提升视频大语言模型，构建首个真实世界音频视觉时空推理数据集，并提出AVST-Zero模型增强推理能力。
摘要：
最近，多模态大语言模型（MLLMs）在视频理解任务中取得了快速进展。然而，当前研究集中在简单的视频场景，未能反映视频中真实世界音频视觉事件的复杂性和多样性。为弥合这一差距，我们首先引入R-AVST，一个具有细粒度时空注解的音频视觉推理数据集。在构建该数据集时，我们设计了一条包含基于LLM的关键对象提取、自动空间注解和人工质量检查的流水线，产生超过5000个未剪辑视频，包含27000个对象，涵盖100种音频视觉事件。基于此数据集，我们定义了三个核心任务，用于音频视觉场景中的时空推理，并生成超过8000对高质量、均匀分布的问题-答案对，以有效评估模型性能。为进一步增强推理能力，我们提出AVST-Zero，一种基于强化学习的模型，避免中间监督，通过精心设计的多维奖励直接优化行为。广泛的实验验证了我们R-AVST在推进音频视觉时空推理方面的有效性，基于此，AVST-Zero在与现有模型的比较中表现出竞争力。据我们所知，R-AVST是首个针对真实世界音频视觉时空推理设计的数据集，而AVST-Zero为该领域未来挑战提供了新的视角。
英文摘要：
Recently, rapid advancements have been made in multimodal large language models (MLLMs), especially in video understanding tasks. However, current research focuses on simple video scenarios, failing to reflect the complex and diverse nature of real-world audio-visual events in videos. To bridge this gap, we firstly introduce R-AVST, a dataset for audio-visual reasoning featuring fine-grained spatio-temporal annotations. In constructing this, we design a pipeline consisting of LLM-based key object extraction, automatic spatial annotation and manual quality inspection, resulting in over 5K untrimmed videos with 27K objects across 100 types of audio-visual events. Building on this dataset, we define three core tasks for spatio-temporal reasoning in audio-visual scenes and generate more than 8K high-quality, evenly distributed question-answer pairs to effectively benchmark model performance. To further enhance reasoning, we propose AVST-Zero, a reinforcement learning-based model that avoids intermediate supervision, directly optimizing behavior via carefully designed multi-dimensional rewards. Extensive experiments validate the effectiveness of our R-AVST in advancing audio-visual spatio-temporal reasoning, upon which AVST-Zero demonstrates competitive performance compared to existing models. To the best of our knowledge, R-AVST is the first dataset designed for real-world audio-visual spatio-temporal reasoning, and AVST-Zero offers a novel perspective for tackling future challenges in this domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
