EVISTA:基于证据感知时间路径优化的统一多模态视频搜索
EVISTA: Unified Multimodal Video Search with Evidence-Aware Temporal Path Optimization
AI总结:
本研究针对多模态视频检索难题,开发了EVISTA系统并提出TEMPO-P方法,在挑战赛查询上实现了多事件的精准有序检索,弥补了仅视觉检索的不足。
摘要:
在大数据时代,从大型视频集合中进行精确的事件检索已变得愈发重要且富有挑战性。尤其是当不同事件由不同类型的证据(如视觉内容、屏幕文本和语音)描述时,按时间顺序定位同一视频中的多个事件十分困难。我们开发了EVISTA——一个面向2026年胡志明市AI挑战赛的统一多模态视频检索系统,该系统在共享关键帧索引上整合了视觉语义、OCR(光学字符识别)、ASR(自动语音识别)、目标及元数据搜索,并提出了TEMPO-P这一用于有序多事件检索的证据感知时间路径优化方法。为该任务构建EVISTA时,我们发现了独立事件检索的两个局限性:当某一事件检索结果较弱时,可能会遗漏相关视频;且独立检索的事件候选可能在时间上不一致,或无法表示主要由文本或语音定义的事件。TEMPO-P通过跨事件和模态聚合候选视频、使用视觉、OCR和ASR证据联合对关键帧评分,并通过动态规划在每个视频内选择紧凑的时间有序路径来解决这些局限性。在5个主办方查询和36个额外的多事件查询上,TEMPO-P在视觉描述事件上达到了仅视觉时间对齐的性能(R@1均为0.42),并能检索包含文本定义事件的查询(R@1为0.38),而仅视觉时间对齐无法表示此类事件。TEMPO-P作为有序事件检索模式被集成到EVISTA中。
英文摘要:
In the era of big data, precise event retrieval from large video collections has become increasingly important and challenging. In particular, locating multiple events from the same video in their temporal order is difficult when different events are described by different types of evidence, such as visual content, on-screen text, and spoken words. We develop EVISTA, a unified multimodal video retrieval system for the Ho Chi Minh City AI Challenge 2026, which integrates visual-semantic, OCR, ASR, object, and metadata search over a shared keyframe index, and propose TEMPO-P, an evidence-aware temporal path optimization method for ordered multi-event retrieval. Building EVISTA for this task reveals two limitations of independent event retrieval: a relevant video may be missed when one event is weakly retrieved, and independently retrieved event candidates may be temporally inconsistent or unable to represent events defined primarily by text or speech. TEMPO-P addresses these limitations by aggregating candidate videos across events and modalities, jointly scoring keyframes using visual, OCR, and ASR evidence, and selecting a compact temporally ordered path within each video through dynamic programming. On five organizer queries and 36 additional multi-event queries, TEMPO-P matches visual-only temporal alignment on visually described events (R@1 0.42 for both) and retrieves queries containing text-defined events, which visual-only alignment cannot represent (R@1 0.38). TEMPO-P is integrated into EVISTA as its ordered-event retrieval mode.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。