3D-PLOT-LLM: 用于三维大语言模型的部件级对象标记
3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models
AI总结：
提出3D-PLOT-LLM，通过重组输入标记流使部件可直接通过LLM词汇寻址，无需分割解码器或边界框，在部件级基准上超越现有方法。
摘要：
三维多模态大语言模型（3D MLLMs）将3D对象作为一个整体进行描述，但无法处理、命名或推理其部件。先前的部件感知尝试增加了分割解码器、更重的3D编码器或边界框语法，导致参数成本大幅增加。我们采取了一条根本不同的路径：重新组织输入标记流，使得部件通过LLM自身的词汇变得可直接寻址。我们的模型3D-PLOT-LLM将冻结的点编码器的块分割成K个局部一致的区域，并在每个区域的块标记之前插入一个可学习的每区域标记和一个保留词汇标记<part_k>；然后，一个标记空间精化（MSR）模块根据每个区域的空间统计信息和邻接邻居对该标记进行条件化。因此，模型在其输出中引用部件，并遵循通过标记引用部件的提示，这是先前对象级3D MLLMs所不具备的能力。为了探究这一接口，我们构建了PartVerse-QA，一个基于PartVerse网格注释改编的词汇级部件问答基准（77K训练对和588个保留查询，基于不相交的对象划分），在该基准上，3D-PLOT-LLM达到了描述到槽的Jaccard指数0.459和精确匹配率13.78%，槽到描述的GPT-4o评判得分为44.68。在3DCoMPaT-GrIn部件感知接地描述基准上，3D-PLOT-LLM在所有文本输出指标上优于PointLLM、Kestrel、PARIS3D和SegPoint，并在4项指标中的3项上优于ShapeLLM，相比PointLLM的GPT-4o评判得分最高提升+3.03。在Objaverse整体对象描述中，在第二阶段添加PartVerse-QA使得相比PointLLM的SBERT得分提升+0.65，GPT-4o得分提升+1.85，并且在5项传统指标中的4项（SBERT、SimCSE、BLEU-1、METEOR）上超过PointLLM-PiSA，尽管其目标是不同的（部件接地）目标。所有这些仅需在冻结的点编码器上增加不到100万个可训练参数，比先前的部件感知3D MLLMs低一个数量级，且无需分割解码器或边界框头。
英文摘要：
3D multimodal large language models (3D MLLMs) describe a 3D object as a whole but cannot address, name, or reason about its parts. Prior part-aware attempts add segmentation decoders, heavier 3D encoders, or bounding-box grammars at substantial parameter cost. We take a fundamentally different path: we reorganize the input token stream so that parts become directly addressable through the LLM's own vocabulary. Our model, 3D-PLOT-LLM, partitions the frozen point encoder's patches into K locally coherent regions and inserts, before each region's patch tokens, a learnable per-region marker and a reserved vocabulary token <part_k>; a Marker-Space Refinement (MSR) module then conditions each marker on its region's spatial statistics and adjacency neighbors. The model thus cites parts in its output and follows prompts that refer to parts by token, a capability absent from prior object-level 3D MLLMs. To probe this interface, we construct PartVerse-QA, a vocabulary-level part-QA benchmark adapted from PartVerse mesh annotations (77K training pairs and 588 held-out queries on disjoint object splits), on which 3D-PLOT-LLM reaches caption-to-slots Jaccard 0.459 and Exact-match 13.78%, with a slot-to-caption GPT-4o judge of 44.68. On the 3DCoMPaT-GrIn part-aware grounded description benchmark, 3D-PLOT-LLM outperforms PointLLM, Kestrel, PARIS3D, and SegPoint on every text-output metric, and ShapeLLM on 3 of 4, with up to +3.03 GPT-4o judge over PointLLM. On Objaverse whole-object captioning, adding PartVerse-QA at Stage 2 yields +0.65 SBERT and +1.85 GPT-4o over PointLLM, and tops PointLLM-PiSA on 4 of 5 traditional metrics (SBERT, SimCSE, BLEU-1, METEOR) despite targeting a different (part-grounded) objective. All with under 1M new trainable parameters on a frozen point encoder, an order of magnitude below prior part-aware 3D MLLMs, and no segmentation decoder or bounding-box head.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
