ExCap3D：通过可变细节的对象字幕实现富有表现力的3D场景理解
ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detail
AI总结：
针对现有3D对象字幕仅单一细节级别的不足，提出ExCap3D模型生成对象级与部件级多细节字幕，构建对应数据集，实验显示其字幕质量显著优于SOTA方法。
摘要：
为3D室内场景中的对象生成文本描述是具身理解的重要组成部分。现有方法仅以单一细节级别描述对象，往往无法捕捉细粒度细节，例如对象各部件的不同纹理、材质和形状。我们提出了富有表现力的3D字幕任务：给定输入3D场景，从多个细节级别描述对象——包括高层对象描述，以及关于其部件属性的低层描述。\n为生成此类字幕，我们提出了ExCap3D这一富有表现力的3D字幕模型。该模型以3D扫描结果为输入，针对扫描中每个检测到的对象，生成其部件的细粒度综合描述，同时生成以部件级描述为条件的对象级描述。我们在设计ExCap3D时鼓励生成的文本描述之间保持语义一致性，并在潜在空间中实现文本相似性，以进一步提升生成字幕的质量。\n为支撑该任务，我们利用视觉语言模型（VLM）进行多视角字幕生成，构建了ExCap3D数据集。该数据集基于ScanNet++数据集生成，包含不同细节级别的字幕，涵盖947个室内场景中34000个3D对象的19万条文本描述。实验表明，ExCap3D生成的对象级和部件级细节字幕质量优于现有最优方法，其中对象级细节的Cider得分提升17%，部件级细节的Cider得分提升124%。我们的代码、数据集和模型将公开发布。
英文摘要：
Generating text descriptions of objects in 3D indoor scenes is an important building block of embodied understanding. Existing methods do this by describing objects at a single level of detail, which often does not capture fine-grained details such as varying textures, materials, and shapes of the parts of objects. We propose the task of expressive 3D captioning: given an input 3D scene, describe objects at multiple levels of detail: a high-level object description, and a low-level description of the properties of its parts. To produce such captions, we present ExCap3D, an expressive 3D captioning model which takes as input a 3D scan, and for each detected object in the scan, generates a fine-grained collective description of the parts of the object, along with an object-level description conditioned on the part-level description. We design ExCap3D to encourage semantic consistency between the generated text descriptions, as well as textual similarity in the latent space, to further increase the quality of the generated captions. To enable this task, we generated the ExCap3D Dataset by leveraging a visual-language model (VLM) for multi-view captioning. The ExCap3D Dataset contains captions on the ScanNet++ dataset with varying levels of detail, comprising 190k text descriptions of 34k 3D objects in 947 indoor scenes. Our experiments show that the object- and part-level of detail captions generated by ExCap3D are of higher quality than those produced by state-of-the-art methods, with a Cider score improvement of 17% and 124% for object- and part-level details respectively. Our code, dataset and models will be made publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
