arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.13035cs.CVcs.AIcs.CL
收藏

GeoGR:用于高效3D视觉语言模型的空间-语义令牌压缩与分组路由

GeoGR: Spatial-Semantic Token Compression and Group Routing for Efficient 3D VLMs

Eastern Institute of Technology(东方理工大学);Shanghai Jiao Tong University(上海交通大学);The Hong Kong Polytechnic University(香港理工大学);LMU Munich(慕尼黑大学)

Yongxin Zheng, Hao Wu, Fang Li, Hu Fu, Peiran Yin, Xudong Wang, Haozhe Hu, Xinghao Chen, Yunpu Ma, Wei Zhang, Xiaoyu Shen

AI总结:

针对3D VLMs推理成本高的问题,提出GeoGR框架,通过GeoSemZip压缩场景表示、GroupRoute优化视觉计算,在保留95%以上性能的同时将LLM预填充速度提升6倍以上。

摘要:

多视图表示使预训练的图像和视频视觉语言模型(VLM)能够理解3D场景,但它们的长视觉令牌序列会导致推理成本高昂。冗余既来自不同视图对同一场景区域的重复观测,也来自在每个大语言模型(LLM)层处理所有视觉令牌,无论其与当前计算的相关性如何。我们提出GeoGR,这是一个两阶段框架,可联合压缩场景表示和视觉计算。在投影器层面,GeoSemZip将跨视图观测整合为3D体素令牌,并结合语义显著性与空间覆盖度来构建紧凑的场景表示;未被选中的体素特征被聚合为上下文锚点,以保留原本会被丢弃的场景信息。在LLM层面,我们确定了一个有效的视觉计算窗口,并观察到查询条件下的令牌相关性在局部保持稳定,但会在较长的层间隔内发生变化。基于这些发现,GroupRoute将晚进入、早退出与可恢复的分组跳跃相结合:低相关性令牌绕过完整的LLM层,同时保留其隐藏状态,并可在计算预算下选定的后续锚点层重新加入活跃组。为实现压缩感知的后训练,我们通过保留原始训练池的30%构建了一个包含6.7万个样本的紧凑数据集。通过该协议,GeoGR在LLaVA-OneVision-7B和Video-3D LLM-7B上分别保留了密集后训练模型95.2%和96.8%的性能,同时将LLM预填充速度提升了6.8倍和6.1倍。这些结果证明了GeoGR在有和无显式3D空间编码的3D VLMs中的一致有效性,凸显了其良好的性能-效率权衡。代码可在该https URL获取。

英文摘要:

Multi-view representations enable pretrained image and video vision-language models to understand 3D scenes, but their long visual token sequences make inference costly. Redundancy arises both from repeated observations of the same scene regions across views and from processing all visual tokens at every LLM layer, regardless of their relevance to the current computation. We propose GeoGR, a two-stage framework that jointly compresses scene representations and visual computation. At the projector level, GeoSemZip consolidates cross-view observations into 3D voxel tokens and combines semantic saliency with spatial coverage to construct a compact scene representation. Unselected voxel features are aggregated into contextual anchors to retain scene information that would otherwise be discarded. At the LLM level, we identify an effective visual computation window and observe that query-conditioned token relevance remains locally stable but changes over longer layer intervals. Guided by these findings, GroupRoute combines late entry and early exit with recoverable group-wise skipping. Low-relevance tokens bypass complete LLM layers while retaining their hidden states and can rejoin the active group at subsequent anchor layers selected under a computation budget. For compression-aware post-training, we construct a compact dataset of 67K samples by retaining 30% of the original training pool. With this protocol, GeoGR retains 95.2% and 96.8% of the dense post-trained models' performance on LLaVA-OneVision-7B and Video-3D LLM-7B, respectively, while accelerating LLM prefill by $6.8\times$ and $6.1\times$. These results demonstrate the consistent effectiveness of GeoGR across 3D VLMs with and without explicit 3D spatial encoding, highlighting a favorable performance-efficiency trade-off. The code is available at https://github.com/EIT-NLP/GeoGR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑