DynaTok: 时序自适应和位置偏见感知的视频大语言模型token压缩
DynaTok: Temporally Adaptive and Positional Bias-Aware Token Compression for Video-LLMs
AI总结：
本文提出DynaTok，一种无需训练的时序自适应和位置偏见感知的token压缩框架，通过在时序和空间维度上分配token预算，有效减少冗余的时空覆盖，提升视频大语言模型的效率和鲁棒性。
摘要：
近年来，视频大语言模型（Video-LLMs）的进步显著扩展了多模态推理能力。然而，从长视频序列中提取的大量视觉token带来了高昂的计算成本，限制了其在现实场景中的应用。现有的无训练token压缩方法基于注意力大小作为语义重要性的代理进行token选择，但往往忽视位置偏见并仅依赖短期时间局部性，导致冗余的时空覆盖和低效的token使用。我们提出了DynaTok，一种无需训练、时序自适应且偏见感知的token压缩框架，能够在时序和空间维度上分配token预算。通过轻量级的指数移动平均（EMA）内存，时序预算分配（TBA）模块动态地将较少的token分配给冗余帧，将更多的token分配给新颖的帧，捕捉长期时间变化。空间预算分配（SBA）模块通过基于激活的注意力图选择空间多样性和语义重要的特征，同时利用空间内存减少已选区域的冗余并缓解位置偏见。DynaTok无缝集成到现有的Video-LLMs中，如LLaVA-OneVision和LLaVA-Video，无需重新训练，并在高强度压缩下有效保留语义覆盖。在四个代表性VideoQA基准测试-MVBench、LongVideoBench、MLVU和VideoMME上的实验表明，即使在90%的token减少下，DynaTok仍能保留超过95%的基线准确性，优于最近的无训练方法。这些结果表明，DynaTok为高效和稳健的视频推理提供了系统的基础，为未来Video-LLMs实现实时流媒体视频理解铺平了道路。
英文摘要：
Recent advances in Video Large Language Models (Video-LLMs) have greatly expanded multimodal reasoning capabilities. However, the massive number of visual tokens extracted from long video sequences incurs prohibitive computational costs, limiting their deployment in real-world scenarios. Existing training-free token compression methods select tokens based on attention magnitude as a proxy for semantic importance, but often overlook positional bias and rely only on short-term temporal locality, leading to redundant spatio-temporal coverage and inefficient token usage. We present DynaTok, a training-free, temporally adaptive and bias-aware token compression framework that allocates token budgets across both temporal and spatial dimensions. Through a lightweight exponential moving average (EMA) memory, the Temporal Budget Allocation (TBA) module dynamically assigns fewer tokens to redundant frames and more to novel frames, capturing long-term temporal variation. The Spatial Budget Allocation (SBA) module complements this by selecting spatially diverse and semantically important features using activation-based attention maps, while leveraging a spatial memory to reduce redundancy from previously selected regions and mitigate positional bias. DynaTok integrates seamlessly with existing Video-LLMs such as LLaVA-OneVision and LLaVA-Video without retraining, and effectively preserves semantic coverage under aggressive compression. Experiments on four representative VideoQA benchmarks-MVBench, LongVideoBench, MLVU, and VideoMME-show that DynaTok retains over 95% of baseline accuracy even with a 90% token reduction, surpassing recent training-free approaches. These results demonstrate that DynaTok provides a principled foundation for efficient and robust video reasoning, paving the way toward real-time streaming video understanding with future Video-LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
