保持证据链:面向视频时序定位的无训练令牌剪枝的语义证据分配
Keeping the Evidence Chain: Semantic Evidence Allocation for Training-Free Token Pruning in Video Temporal Grounding
AI总结:
提出SemVID无训练剪枝框架,通过证据保留和连接强度原则选择对象、运动和上下文令牌,在仅保留12.5%视觉令牌时保持95.4% mIoU,实现5.8倍预填充加速。
摘要:
视频时序定位(VTG)在长视频中定位查询相关时刻的时间边界,使得视频-语言模型计算成本极高。虽然最近的无训练令牌剪枝在视频问答中取得了成功,但将这些目标简单应用于VTG会导致性能急剧下降,因为VTG关键依赖于边界敏感的证据和跨帧推理链。因此,我们确定了两个VTG特定的剪枝原则:证据保留(保留查询关键补丁)和连接强度(保留跨帧连接以进行长距离证据聚合)。我们提出了SemVID,一个无训练剪枝框架,构建一个紧凑且连贯的令牌子集,具有互补的语义角色。SemVID首先通过平衡查询相关性和帧间变化来分配每帧预算,以避免过度剪枝片段,然后选择三种类型的令牌:用于多样化查询关键证据的对象令牌、用于捕获过渡并作为跨帧中继的运动令牌,以及用于场景连续性的上下文令牌。大量实验表明,SemVID实现了强大的精度-效率权衡,仅使用12.5%的视觉令牌即可保留高达95.4%的mIoU,并提供高达5.8倍的预填充加速。我们的代码可在以下网址获取:https://this URL
英文摘要:
Video Temporal Grounding (VTG) localizes the temporal boundaries of query-relevant moments in long, untrimmed videos, making video-language-model prohibitively expensive. While recent training-free token pruning has shown success in video question answering, naively applying these objectives to VTG causes drastic degradation, as VTG crucially depends on boundary-sensitive evidence and cross-frame reasoning chains. We therefore identify two VTG-specific pruning principles: evidence retention, which keeps query-critical patches especially around event boundaries, and connectivity strength, which preserves cross-frame connectivity for long-range evidence aggregation. Building on these insights, we propose SemVID, a training-free pruning framework that constructs a compact yet coherent token subset with complementary semantic roles. SemVID first allocates per-frame budgets by balancing query relevance and inter-frame variation to avoid over-pruned segments, and then selects three types of tokens: object tokens for diverse query-critical evidence, motion tokens to capture meaningful transitions and serve as cross-frame relays, and context tokens for scene continuity. Extensive experiments show that SemVID achieves a strong accuracy-efficiency trade-off, retaining up to 95.4% mIoU with only 12.5% visual tokens and delivering up to a 5.8x prefill speedup, consistently outperforming prior methods under the same budgets. Our code is available at https://github.com/JiaqiLi404/SemVID
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。