HCC-3D:用于视觉语言模型中98%三维Token压缩的层级补偿压缩方法
HCC-3D: Hierarchical Compensatory Compression for 98% 3D Token Reduction in Vision-Language Models
AI总结:
针对现有3D-VLM因LLM需处理全部3D Token导致计算成本过高的问题,提出HCC-3D层级补偿压缩方法,实现约98%的3D Token压缩率并取得新SOTA性能。
摘要:
近年来,三维理解领域受到广泛关注,相关研究利用视觉语言模型(VLMs)实现点云与文本数据间的多模态推理。当前的3D-VLM沿用具备强大推理能力的大型2D-VLM范式,直接将三维点云嵌入为3D Token,但该框架计算成本极高,限制了其实际应用;我们经分析发现,其瓶颈在于大语言模型(LLM)部分需要处理全部3D Token。由此引出核心问题:如何在保留3D Token关键信息完整性的前提下,降低其带来的计算开销?为解决该问题,我们提出层级补偿压缩方法HCC-3D,可在高效压缩3D Token的同时保留关键细节。具体而言,我们首先提出全局结构压缩(GSC)模块,通过设计全局查询将全部3D Token压缩为少量关键Token,同时保留整体结构信息;随后,为弥补GSC造成的信息损失,我们进一步提出自适应细节挖掘(ADM)模块,通过互补评分选择性地对显著但未被充分关注的特征进行重压缩。大量实验表明,与此前的3D-VLM相比,HCC-3D不仅实现了约98%的极高压缩率,还取得了新的最优性能,在效率和性能两方面均展现出显著提升。
英文摘要:
3D understanding has drawn significant attention recently, leveraging Vision-Language Models (VLMs) to enable multi-modal reasoning between point cloud and text data. Current 3D-VLMs directly embed the 3D point clouds into 3D tokens, following large 2D-VLMs with powerful reasoning capabilities. However, this framework has a great computational cost limiting its application, where we identify that the bottleneck lies in processing all 3D tokens in the Large Language Model (LLM) part. This raises the question: how can we reduce the computational overhead introduced by 3D tokens while preserving the integrity of their essential information? To address this question, we introduce Hierarchical Compensatory Compression (HCC-3D) to efficiently compress 3D tokens while maintaining critical detail retention. Specifically, we first propose a global structure compression (GSC), in which we design global queries to compress all 3D tokens into a few key tokens while keeping overall structural information. Then, to compensate for the information loss in GSC, we further propose an adaptive detail mining (ADM) module that selectively recompresses salient but under-attended features through complementary scoring. Extensive experiments demonstrate that HCC-3D not only achieves extreme compression ratios (approximately 98%) compared to previous 3D-VLMs, but also achieves new state-of-the-art performance, showing the great improvements on both efficiency and performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。