看得更多，说得更多：轻量级语言专家作为动态视频令牌压缩器
Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors
AI总结：
提出LangDC，一种语言感知的动态视频令牌压缩器，利用轻量级语言模型生成软字幕令牌，根据场景丰富度动态调整压缩比，减少49% FLOPs并保持性能。
摘要：
近期大型视频语言模型的进展彻底改变了视频理解任务。然而，其效率受到处理大量视觉令牌的显著限制。现有的令牌压缩策略采用固定压缩比，忽略了不同视频片段之间语义密度的差异性。因此，这导致信息丰富的片段因令牌不足而表示不充分，同时对静态或内容贫乏的片段进行不必要的计算。为解决这一问题，我们提出了LangDC，一种语言感知的动态令牌压缩器。LangDC利用轻量级语言模型描述视频片段，将其转换为软字幕令牌作为视觉表示。通过我们提出的语义密度感知监督进行训练，LangDC旨在1）覆盖下游任务推理所需的关键视觉线索，2）根据场景丰富度（通过描述长度反映）动态调整压缩比。我们的设计模仿了人类动态表达所见内容的方式：复杂场景（看得更多）引发更详细的语言以传达细微差别（说得更多），而简单场景则用更少的词描述。实验结果表明，与VideoGPT+相比，我们的方法将FLOPs减少了49%，同时保持了有竞争力的性能。此外，定性结果证明我们的方法能根据视频片段的丰富度自适应调整令牌压缩比。
英文摘要：
Recent advancements in large video-language models have revolutionized video understanding tasks. However, their efficiency is significantly constrained by processing high volumes of visual tokens. Existing token compression strategies apply a fixed compression ratio, ignoring the variability in semantic density among different video clips. Consequently, this lead to inadequate representation of information-rich clips due to insufficient tokens and unnecessary computation on static or content-poor ones. To address this, we propose LangDC, a Language-aware Dynamic Token Compressor. LangDC leverages a lightweight language model to describe video clips, converting them into soft caption tokens as visual representations. Trained with our proposed semantic density-aware supervision, LangDC aims to 1) cover key visual cues necessary for downstream task reasoning and 2) dynamically adjust compression ratios based on scene richness, reflected by descriptions length. Our design mimics how humans dynamically express what they see: complex scenes (seeing more) elicit more detailed language to convey nuances (saying more), whereas simpler scenes are described with fewer words. Experimental results show that our method reduces FLOPs by 49% compared to VideoGPT+ while maintaining competitive performance. Furthermore, qualitative results demonstrate our approach adaptively adjusts the token compression ratio based on video segment richness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
