发表机构
Beijing University of Posts and Telecommunications（北京邮电大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究探究低成本自监督点云编码器能否替代昂贵多模态编码器用于3D-LLM，经实验发现其与架构存在强交互，为高性价比3D-LLM设计提供指南。
AI 中文摘要
3D点云-语言模型（3D-LLMs）通过将点云编码器与大语言模型配对实现3D理解，但现有方法依赖昂贵的多模态编码器（如ULIP-2），需在8块A100规模的计算资源上完成图像-文本-点云对齐，给研究和部署带来高门槛。本研究系统探究低成本自监督点云编码器（即PCP-MAE和Point-MAE）是否可作为有效替代方案。以MiniGPT-3D为测试平台，在冻结和非冻结微调（共12组）下，评估7种编码器初始化/预训练设置（1种多模态基线、5种自监督、1种随机初始化），涉及2种架构（MaskTransformer、PointTransformer）、3种目标（PCP-MAE、Point-MAE、随机初始化）和2种数据集（Objaverse 660K、ShapeNet55-34约5万）。实验得出三个关键发现：（1）四阶段MiniGPT-3D pipeline可有效从随机初始化训练3D编码器，端到端训练的随机初始化编码器达到52.50%开放词汇准确率和44.45%字幕生成分数，接近顶级预训练变体；（2）架构与预训练目标存在强交叉交互：PCP-MAE+MaskTransformer达到59.00%准确率（最佳自监督），而Point-MAE+MaskTransformer降至46.50%，PointTransformer则呈现相反模式；（3）纯几何编码器的闭集ModelNet40分类仍是核心短板，仅达到约13-18%准确率，而多模态基线约为62%，即便经过端到端微调。研究结果为高性价比3D-LLM设计提供实用指南，并揭示自监督目标与编码器架构间的交互模式。
英文摘要
3D point cloud-language models (3D-LLMs) enable 3D understanding by pairing point cloud encoders with large language models, but existing methods rely on costly multi-modal encoders (e.g., ULIP-2) that require image-text-point cloud alignment on 8x A100-scale compute, creating high barriers for research and deployment. In this work, we systematically investigate whether low-cost self-supervised point cloud encoders, specifically PCP-MAE and Point-MAE, can serve as effective alternatives. Using MiniGPT-3D as our testbed, we evaluate 7 encoder initialization/pre-training setups (1 multi-modal baseline, 5 self-supervised, 1 random init) under frozen and unfrozen fine-tuning (12 total groups), across 2 architectures (MaskTransformer, PointTransformer), 3 objectives (PCP-MAE, Point-MAE, random init), and 2 datasets (Objaverse 660K, ShapeNet55-34 approximately 50K). Our experiments reveal three key findings: (1) The four-stage MiniGPT-3D pipeline can effectively train a 3D encoder from random initialization: an end-to-end trained random init encoder reaches 52.50% open-vocabulary accuracy and 44.45 captioning score, approaching top pre-trained variants; (2) Architecture and pre-training objective show strong crossover interaction: PCP-MAE + MaskTransformer achieves 59.00% accuracy (best self-supervised), while Point-MAE + MaskTransformer drops to 46.50%, with the pattern reversed for PointTransformer; (3) Closed-set ModelNet40 classification remains a core weakness of purely geometric encoders, reaching only ~13-18% accuracy vs. ~62% for the multi-modal baseline, even after end-to-end fine-tuning. Our results offer practical guidelines for cost-effective 3D-LLM design and reveal interaction patterns between self-supervised objectives and encoder architectures.
Comments14 pages, 3 figures. This work has been previously released as a preprint on ChinaXiv (No. ChinaXiv:202607.00167, DOI: 10.12074/202607.00167)
