FreeOcc: 通过基础模型实现无训练的全景占用预测
FreeOcc: Training-free Panoptic Occupancy Prediction via Foundation Models
AI总结：
FreeOcc通过基础模型实现无训练的全景占用预测，无需学习3D模型即可生成实例感知的3D场景表示。
摘要：
语义和全景占用预测用于道路场景分析，为自动驾驶车辆的周围环境提供密集的3D表示。当前仅依靠摄像头的方法通常依赖于成本高昂的密集3D监督或需要在目标领域数据上训练模型，限制了在未见环境中的部署。我们提出了FreeOcc，一种无训练的流程，利用预训练的基础模型从多视角图像中恢复语义和几何信息。FreeOcc通过可提示的基础分割模型提取每视角的全景先验，并通过提示到分类规则进行提取，然后使用重建基础模型重建度量3D点。深度和置信度意识的过滤将可靠的标签提升到3D，这些标签在时间上融合并用确定性细化堆栈进行体素化。对于全景占用，实例通过拟合和合并稳健的当前视角3D框候选物进行恢复，从而在不使用任何学习的3D模型的情况下实现实例感知的占用。在Occ3D-nuScenes上，FreeOcc实现了16.9 mIoU和16.5 RayIoU无训练结果，与最先进的弱监督方法相当。当作为生成伪标签的流程用于训练下游模型时，它实现了21.1 RayIoU，超过了之前的最先进弱监督基线。此外，FreeOcc在无训练和弱监督的全景占用预测中设定了新的基准，分别实现了3.1 RayPQ和3.9 RayPQ。这些结果突显了基础模型驱动的感知作为无训练3D场景理解的实用途径。
英文摘要：
Semantic and panoptic occupancy prediction for road scene analysis provides a dense 3D representation of the ego vehicle's surroundings. Current camera-only approaches typically rely on costly dense 3D supervision or require training models on data from the target domain, limiting deployment in unseen environments. We propose FreeOcc, a training-free pipeline that leverages pretrained foundation models to recover both semantics and geometry from multi-view images. FreeOcc extracts per-view panoptic priors with a promptable foundation segmentation model and prompt-to-taxonomy rules, and reconstructs metric 3D points with a reconstruction foundation model. Depth- and confidence- aware filtering lifts reliable labels into 3D, which are fused over time and voxelized with a deterministic refinement stack. For panoptic occupancy, instances are recovered by fitting and merging robust current-view 3D box candidates, enabling instance-aware occupancy without any learned 3D model. On Occ3D-nuScenes, FreeOcc achieves 16.9 mIoU and 16.5 RayIoU train-free, on par with state-of-the-art weakly supervised methods. When employed as a pseudo-label generation pipeline for training downstream models, it achieves 21.1 RayIoU, surpassing the previous state-of-the-art weakly supervised baseline. Furthermore, FreeOcc sets new baselines for both train-free and weakly supervised panoptic occupancy prediction, achieving 3.1 RayPQ and 3.9 RayPQ, respectively. These results highlight foundation-model-driven perception as a practical route to training-free 3D scene understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
