PartDistill：通过视觉-语言模型蒸馏实现3D形状部件分割
PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillation
AI总结：
本文提出跨模态蒸馏框架 PartDistill，利用 VLM 教师与3D学生的双向蒸馏及生成式知识源，显著提升 ShapeNetPart 和 PartNetE 上的3D部件分割性能。
摘要：
本文提出了一个跨模态蒸馏框架 PartDistill，它迁移来自视觉-语言模型（VLM）的2D知识，以促进3D形状部件分割。PartDistill 解决了该任务中的三大挑战：2D投影中不可见或未检测区域缺少3D分割、VLM给出的2D预测不一致，以及不同3D形状之间缺乏知识积累。PartDistill 由一个教师网络和一个学生网络组成：教师网络使用 VLM 进行2D预测，学生网络从这些2D预测中学习，同时从多个3D形状中提取几何特征以执行3D部件分割。该框架内实施了包含前向蒸馏和后向蒸馏的双向蒸馏，其中前者将2D预测蒸馏给学生网络，后者提升2D预测的质量，随后增强最终的3D分割。此外，PartDistill 可以利用生成模型来轻松创建3D形状，从而生成待蒸馏的知识来源。通过大量实验，PartDistill 在广泛使用的 ShapeNetPart 和 PartNetE 数据集上以显著优势提升了现有方法，mIoU 分数分别提高超过15%和12%。本工作的代码可在 https://github.com/ardianumam/PartDistill 获取。
英文摘要：
This paper proposes a cross-modal distillation framework, PartDistill, which transfers 2D knowledge from vision-language models (VLMs) to facilitate 3D shape part segmentation. PartDistill addresses three major challenges in this task: the lack of 3D segmentation in invisible or undetected regions in the 2D projections, inconsistent 2D predictions by VLMs, and the lack of knowledge accumulation across different 3D shapes. PartDistill consists of a teacher network that uses a VLM to make 2D predictions and a student network that learns from the 2D predictions while extracting geometrical features from multiple 3D shapes to carry out 3D part segmentation. A bi-directional distillation, including forward and backward distillations, is carried out within the framework, where the former forward distills the 2D predictions to the student network, and the latter improves the quality of the 2D predictions, which subsequently enhances the final 3D segmentation. Moreover, PartDistill can exploit generative models that facilitate effortless 3D shape creation for generating knowledge sources to be distilled. Through extensive experiments, PartDistill boosts the existing methods with substantial margins on widely used ShapeNetPart and PartNetE datasets, by more than 15% and 12% higher mIoU scores, respectively. The code for this work is available at https://github.com/ardianumam/PartDistill.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
