arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.29437cs.CV
收藏

SeGPruner: 用于3D问答的语义-几何视觉令牌修剪器

SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answering

Shanghai University(上海大学)

Wenli Li, Kai Zhao, Haoran Jiang, Enquan Yang, Yi Su, Dan Zeng

AI总结:

本文提出SeGPruner,通过语义和几何引导的令牌减少框架提升3D问答效率,显著降低视觉令牌预算和推理延迟,同时保持竞争性性能。

摘要:

视觉语言模型(VLMs)已被广泛用于3D问答(3D QA)。在典型流程中,从多个视角提取的视觉令牌与语言令牌连接,并通过大语言模型(LLM)联合处理以进行推断。然而,聚合多视角观察不可避免地引入严重的令牌冗余,导致过于庞大的视觉令牌集,显著阻碍在受限令牌预算下的推理效率。视觉令牌修剪已成为解决此问题的普遍策略。然而,大多数现有修剪器主要针对2D输入或依赖间接几何线索,限制了它们保留语义关键对象和维持足够空间覆盖以实现稳健3D推理的能力。在本文中,我们提出SeGPruner,一种语义感知和几何引导的令牌减少框架,用于高效3D QA多视角图像。具体而言,SeGPruner首先通过基于注意力的重要性模块(语义感知令牌选择器)保留语义显著的令牌,确保对象关键证据得以保留。然后,通过几何引导选择器(几何感知令牌多样化器)补充这些令牌,该选择器共同考虑语义相关性和3D几何距离。这种在语义保留和几何引导多样化之间的合作在剧烈令牌减少下平衡了对象级证据和全局场景覆盖。在ScanQA和OpenEQA上的大量实验表明,SeGPruner显著提高了推理效率,将视觉令牌预算减少91%,推理延迟减少86%,同时在3D推理任务中保持竞争性性能。

英文摘要:

Vision-language models (VLMs) have been widely adopted for 3D question answering (3D QA). In typical pipelines, visual tokens extracted from multiple viewpoints are concatenated with language tokens and jointly processed by a large language model (LLM) for inference. However, aggregating multi-view observations inevitably introduces severe token redundancy, leading to an overly large visual token set that significantly hinders inference efficiency under constrained token budgets. Visual token pruning has emerged as a prevalent strategy to address this issue. Nevertheless, most existing pruners are primarily tailored to 2D inputs or rely on indirect geometric cues, which limits their ability to explicitly retain semantically critical objects and maintain sufficient spatial coverage for robust 3D reasoning. In this paper, we propose SeGPruner, a semantic-aware and geometry-guided token reduction framework for efficient 3D QA with multi-view images. Specifically, SeGPruner first preserves semantically salient tokens through an attention-based importance module (Saliency-aware Token Selector), ensuring that object-critical evidence is retained. It then complements these tokens with spatially diverse ones via a geometry-guided selector (Geometry-aware Token Diversifier), which jointly considers semantic relevance and 3D geometric distance. This cooperation between saliency preservation and geometry-guided diversification balances object-level evidence and global scene coverage under aggressive token reduction. Extensive experiments on ScanQA and OpenEQA demonstrate that SeGPruner substantially improves inference efficiency, reducing the visual token budget by 91% and inference latency by 86%, while maintaining competitive performance in 3D reasoning tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑