arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1709.09843cs.CV
收藏

多模态场景分析中的软对应

Soft Correspondences in Multimodal Scene Parsing

CSIRO (DATA61)(澳大利亚联邦科学与工业研究组织(数据61));Australian National University(澳大利亚国立大学);EPFL(洛桑联邦理工学院)

Sarah Taghavi Namin, Mohammad Najafi, Mathieu Salzmann, Lars Petersson

摘要:

利用多种模态进行语义场景解析已被证明可以提高单模态场景的准确性。然而,多模态数据集往往遭受的问题,如数据错位和标签不一致,其中现有的方法假设,在两个模态中的相应区域必须有相同的标签。我们建议解决这个问题,通过制定多模态语义标记作为CRF中的推理,并引入潜在节点来明确地模拟两种模态之间的不一致。这些潜在节点不仅使我们能够利用来自两个域的信息来改进它们的标记,而且还可以切割不一致区域之间的边缘。我们建议从训练数据中学习域内和域间势函数,以避免手动调整模型参数。我们在两个公开的包含2D和3D数据的数据集上评估了我们的方法。由于我们的潜在节点和我们的学习策略,我们的方法在这两种情况下都优于最先进的方法。此外,为了突出几何信息的好处和我们的方法在同时进行2D/3D语义和几何推理中的潜力,我们在2D和3D中同时进行语义和几何类的推理,从而使两个数据集中的标记结果得到令人满意的改善。

英文摘要:

Exploiting multiple modalities for semantic scene parsing has been shown to improve accuracy over the singlemodality scenario. However multimodal datasets often suffer from problems such as data misalignment and label inconsistencies, where the existing methods assume that corresponding regions in two modalities must have identical labels. We propose to address this issue, by formulating multimodal semantic labeling as inference in a CRF and introducing latent nodes to explicitly model inconsistencies between two modalities. These latent nodes allow us not only to leverage information from both domains to improve their labeling, but also to cut the edges between inconsistent regions. We propose to learn intradomain and inter-domain potential functions from training data to avoid hand-tuning of the model parameters. We evaluate our approach on two publicly available datasets containing 2D and 3D data. Thanks to our latent nodes and our learning strategy, our method outperforms the state-of-the-art in both cases. Moreover, in order to highlight the benefits of the geometric information and the potential of our method in simultaneous 2D/3D semantic and geometric inference, we performed simultaneous inference of semantic and geometric classes both in 2D and 3D that led to satisfactory improvements of the labeling results in both datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑