LongVALE：面向长视频时间感知全模态感知的视觉-音频-语言-事件基准
LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos
AI总结：
针对现有视频理解任务粗粒度、单模态及标注数据不足的问题，研究提出自动化构建流程，发布了含8.4K长视频、105K带精确时间边界和关系感知字幕的全模态事件基准LongVALE，并构建基线验证了其在推动视频大模型全模态细粒度时间理解上的有效性。
摘要：
尽管视频理解领域已取得显著进展，但多数研究仍局限于粗粒度或仅视觉的视频任务。然而，现实世界的视频包含全模态信息（视觉、音频和语音），由一系列事件构成连贯的故事线。缺乏带有细粒度事件标注的多模态视频数据以及人工标注成本高昂，是实现全面全模态视频感知的主要障碍。为填补这一空白，我们提出了一套自动化流程，包含高质量多模态视频筛选、语义连贯的全模态事件边界检测，以及跨模态相关性感知的事件字幕生成。通过该流程，我们构建了LongVALE——首个视觉-音频-语言事件理解基准，涵盖8.4K个高质量长视频中的105K个全模态事件，这些事件具备精确的时间边界和详细的关系感知字幕。此外，我们构建了一个基线模型，首次利用LongVALE使视频大语言模型（LLMs）能够实现全模态细粒度时间视频理解。大量实验证明，LongVALE在推动全面多模态视频理解方面具备有效性与巨大潜力。
英文摘要：
Despite impressive advancements in video understanding, most efforts remain limited to coarse-grained or visual-only video tasks. However, real-world videos encompass omni-modal information (vision, audio, and speech) with a series of events forming a cohesive storyline. The lack of multi-modal video data with fine-grained event annotations and the high cost of manual labeling are major obstacles to comprehensive omni-modality video perception. To address this gap, we propose an automatic pipeline consisting of high-quality multi-modal video filtering, semantically coherent omni-modal event boundary detection, and cross-modal correlation-aware event captioning. In this way, we present LongVALE, the first-ever Vision-Audio-Language Event understanding benchmark comprising 105K omni-modal events with precise temporal boundaries and detailed relation-aware captions within 8.4K high-quality long videos. Further, we build a baseline that leverages LongVALE to enable video large language models (LLMs) for omni-modality fine-grained temporal video understanding for the first time. Extensive experiments demonstrate the effectiveness and great potential of LongVALE in advancing comprehensive multi-modal video understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
