Mantis:Mamba原生微调在3D点云基础模型中的高效性
Mantis: Mamba-native Tuning is Efficient for 3D Point Cloud Foundation Models
AI总结:
针对Mamba架构基础模型的微调问题,提出Mantis框架,通过引入状态感知适配器和双序列一致性蒸馏,实现高效且稳定的微调,仅需约5%的可训练参数。
摘要:
预训练的3D点云基础模型(PFMs)在多种下游任务中表现出强大的迁移能力。然而,对这些模型进行全微调计算成本高且存储需求大。参数高效微调(PEFT)提供了一种有前途的替代方案,但现有PEFT方法主要针对Transformer架构,并依赖于token级提示或特征转换。Mamba架构引入了token级适应与state级序列动态之间的粒度不匹配。因此,将现有PEFT方法直接转移到冻结的Mamba架构上会导致显著的精度下降和不稳定的优化。为了解决这一问题,我们提出了Mantis,这是首个针对3D PFMs的Mamba原生PEFT框架。具体而言,引入了状态感知适配器(SAA),将轻量级任务条件控制信号注入到选择性状态空间更新中,从而实现state级适应,同时保持预训练主干冻结。此外,不同的有效点云序列化通过双序列一致性蒸馏(DSCD)进行正则化,从而减少序列化引起的不稳定性。在多个基准测试中,我们的Mantis在仅约5%的可训练参数下实现了具有竞争力的性能。我们的代码可在https://github.com/gzhhhhhhh/Mantis上获得。
英文摘要:
Pre-trained 3D point cloud foundation models (PFMs) have demonstrated strong transferability across diverse downstream tasks. However, full fine-tuning these models is computationally expensive and storage-intensive. Parameter-efficient fine-tuning (PEFT) offers a promising alternative, but existing PEFT approaches are primarily designed for Transformer-based backbones and rely on token-level prompting or feature transformation. Mamba-based backbones introduce a granularity mismatch between token-level adaptation and state-level sequence dynamics. Consequently, straightforward transfer of existing PEFT approaches to frozen Mamba backbones leads to substantial accuracy degradation and unstable optimization. To address this issue, we propose Mantis, the first Mamba-native PEFT framework for 3D PFMs. Specifically, a State-Aware Adapter (SAA) is introduced to inject lightweight task-conditioned control signals into selective state-space updates, enabling state-level adaptation while keeping the pre-trained backbone frozen. Moreover, different valid point cloud serializations are regularized by Dual-Serialization Consistency Distillation (DSCD), thereby reducing serialization-induced instability. Extensive experiments across multiple benchmarks demonstrate that Mantis achieves competitive performance while training less than 5\% of the backbone parameters.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。