arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.21780cs.CV
收藏

XStreamVGGT: 极端内存高效的流式视觉几何 grounded Transformer 与 KV 缓存压缩

XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression

Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Institute of Artificial Intelligence, Xiamen University(厦门大学人工智能学院);China Star Optoelectronics Technology(中国星电科技);TCL Corporate Research (HK) Co., Ltd.(TCL香港企业研究有限公司);Department of Electrical and Electronic Engineering, The University of Hong Kong(香港大学电子与电气工程系)

Zunhai Su, Weihao Ye, Hansen Feng, Keyu Fan, Jing Zhang, Dahai Yu, Zhengwu Liu, Ngai Wong

AI总结:

XStreamVGGT通过剪枝和量化技术,实现极内存高效的流式3D重建,显著降低内存消耗并加速推理。

摘要:

基于学习的3D视觉几何模型随着大规模Transformer的出现显著进步。其中,StreamVGGT利用帧级因果注意力来提供稳健且高效的流式3D重建。然而,由于多图像和长视频输入带来的大量视觉token流入,导致Key-Value(KV)缓存无限制增长,从而导致内存消耗和推理延迟随着输入帧的累积而增加。这最终限制了其在长时间范围应用中的可扩展性。为了解决这一差距,我们提出了XStreamVGGT,一种无需调优的方法,它无缝集成了剪枝和量化,以系统性地压缩KV缓存,从而实现极内存高效的流式推理。具体而言,从多帧输入生成的冗余KVs首先通过高效的token重要性识别机制剪枝,以符合固定的KV内存预算,同时保持与高性能注意力内核(例如FlashAttention)的完全兼容性。此外,利用KV张量固有的分布模式,我们在剪枝流程中应用维度自适应的KV量化,以进一步减少内存开销,同时保持数值精度。广泛的评估显示,XStreamVGGT在性能上几乎没有下降,同时显著减少了内存使用(减少4.42倍)并加速了推理(加速5.48倍),使实际可扩展的流式3D应用成为可能。代码可在https://github.com/ywh187/XStreamVGGT/上获得。

英文摘要:

Learning-based 3D visual geometry models have significantly advanced with the advent of large-scale transformers. Among these, StreamVGGT leverages frame-wise causal attention to deliver robust and efficient streaming 3D reconstruction. However, it suffers from unbounded growth in the Key-Value (KV) cache due to the massive influx of vision tokens from multi-image and long-video inputs, leading to increased memory consumption and inference latency as input frames accumulate. This ultimately limits its scalability for long-horizon applications. To address this gap, we propose XStreamVGGT, a tuning-free approach that seamlessly integrates pruning and quantization to systematically compress the KV cache, enabling extremely memory-efficient streaming inference. Specifically, redundant KVs generated from multi-frame inputs are initially pruned to conform to a fixed KV memory budget using an efficient token-importance identification mechanism that maintains full compatibility with high-performance attention kernels (e.g., FlashAttention). Additionally, leveraging the inherent distribution patterns of KV tensors, we apply dimension-adaptive KV quantization within the pruning pipeline to further minimize memory overhead while preserving numerical accuracy. Extensive evaluations show that XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage by 4.42$\times$ and accelerating inference by 5.48$\times$, enabling practical and scalable streaming 3D applications. The code is available at https://github.com/ywh187/XStreamVGGT/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑