在表征间切换：面向多模态语义的图像、点云与网格之间的信息传递研究
Juggling With Representations: On the Information Transfer Between Imagery, Point Clouds, and Meshes for Multi-Modal Semantics
AI总结：
该研究提出一种以网格为核心的几何驱动多模态融合方法，通过特征与标签传递实现图像、点云、网格间的信息共享，可降低标注成本并支持多/单模态语义分割，在两个数据集上验证了有效性。
摘要：
过去十年间，对海量已获取的遥感数据进行自动语义分割已成为一项重要任务。图像与点云（Point Clouds，PCs）是基础的数据表征形式，在城市测绘应用中尤为如此。带纹理的三维网格通过将点云进行网格连接、并用可用图像为表面元素赋予纹理，从几何层面整合了这两种数据表征形式。\n我们提出了一种以网格为核心、由几何驱动的整体性方法，可显式整合图像、点云和网格三类实体。鉴于其整合特性，我们选择网格作为核心表征形式，这也有助于解决图像中点的可见性问题。以所提出的多模态融合为基础框架，并结合已建立的实体关联关系，我们通过两种方式实现图像、点云和网格这三类模态间的信息共享：（i）特征传递与（ii）标签传递。\n通过这些方式，我们成功将每类表征的特征向量丰富为多模态特征向量。同时，我们实现了所有表征的一致性标注，还将人工标注工作量降低至仅需对单种表征进行标注。由此，我们为机器学习算法的训练以及任意一类数据表征的语义分割提供了便利——无论是多模态还是单模态场景下均可适用。\n本文介绍了关联机制及后续的信息传递过程，我们认为这两者是多模态场景分析的基石。此外，我们还详细讨论了所提方法的前提条件与局限性。我们在ISPRS三维语义标注竞赛数据集（Vaihingen 3D）以及一个自有数据集（Hessigheim 3D）上验证了该方法的有效性。
英文摘要：
The automatic semantic segmentation of the huge amount of acquired remote sensing data has become an important task in the last decade. Images and Point Clouds (PCs) are fundamental data representations, particularly in urban mapping applications. Textured 3D meshes integrate both data representations geometrically by wiring the PC and texturing the surface elements with available imagery. We present a mesh-centered holistic geometry-driven methodology that explicitly integrates entities of imagery, PC and mesh. Due to its integrative character, we choose the mesh as the core representation that also helps to solve the visibility problem for points in imagery. Utilizing the proposed multi-modal fusion as the backbone and considering the established entity relationships, we enable the sharing of information across the modalities imagery, PC and mesh in a two-fold manner: (i) feature transfer and (ii) label transfer. By these means, we achieve to enrich feature vectors to multi-modal feature vectors for each representation. Concurrently, we achieve to label all representations consistently while reducing the manual label effort to a single representation. Consequently, we facilitate to train machine learning algorithms and to semantically segment any of these data representations - both in a multi-modal and single-modal sense. The paper presents the association mechanism and the subsequent information transfer, which we believe are cornerstones for multi-modal scene analysis. Furthermore, we discuss the preconditions and limitations of the presented approach in detail. We demonstrate the effectiveness of our methodology on the ISPRS 3D semantic labeling contest (Vaihingen 3D) and a proprietary data set (Hessigheim 3D).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
