面向自动驾驶中高效多模态大语言模型的视频Token稀疏化
Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Driving
AI总结：
针对自动驾驶中多模态大语言模型视觉token多、计算与内存开销大的问题，提出视频Token稀疏化方法VTS，通过轻量CNN模型自适应选关键帧并剪枝冗余token，在基准测试中实现了推理吞吐量提升与内存占用降低，且不损失性能。
摘要：
多模态大语言模型（MLLMs）凭借强大的逻辑推理能力，在提升自动驾驶系统的场景理解方面展现出显著潜力。然而，这类模型参数量庞大、计算需求高，往往超出车载计算能力的限制，其部署面临严峻挑战。一大局限在于，捕获细粒度长上下文视觉信息需要大量视觉token，导致延迟升高、内存消耗增加。为解决该问题，我们提出视频Token稀疏化（Video Token Sparsification，VTS）——一种利用连续视频帧固有冗余的新方法，可在保留最显著信息的同时大幅减少视觉token总数。VTS采用基于轻量CNN的候选框模型，自适应识别关键帧并剪枝信息量较低的token，在不损失性能的前提下有效缓解幻觉问题、提升推理吞吐量。我们在DRAMA和LingoQA基准上开展了全面实验，结果表明，与基线相比，VTS可实现最高33%的推理吞吐量提升和28%的内存占用降低，且性能不受影响。
英文摘要：
Multimodal large language models (MLLMs) have demonstrated remarkable potential for enhancing scene understanding in autonomous driving systems through powerful logical reasoning capabilities. However, the deployment of these models faces significant challenges due to their substantial parameter sizes and computational demands, which often exceed the constraints of onboard computation. One major limitation arises from the large number of visual tokens required to capture fine-grained and long-context visual information, leading to increased latency and memory consumption. To address this issue, we propose Video Token Sparsification (VTS), a novel approach that leverages the inherent redundancy in consecutive video frames to significantly reduce the total number of visual tokens while preserving the most salient information. VTS employs a lightweight CNN-based proposal model to adaptively identify key frames and prune less informative tokens, effectively mitigating hallucinations and increasing inference throughput without compromising performance. We conduct comprehensive experiments on the DRAMA and LingoQA benchmarks, demonstrating the effectiveness of VTS in achieving up to a 33\% improvement in inference throughput and a 28\% reduction in memory usage compared to the baseline without compromising performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
