面向3D视觉定位的点-语言层级对齐学习
Learning Point-Language Hierarchical Alignment for 3D Visual Grounding
AI总结：
该研究提出层级对齐模型HAM，通过带上下文调制的点-语言对齐机制和空间多粒度建模实现多粒度视觉语言表征对齐，在3D视觉定位任务上取得SOTA并获ECCV 2022 ScanRefer挑战赛冠军。
摘要：
本文提出一种新型层级对齐模型（Hierarchical Alignment Model，HAM），以端到端方式学习多粒度视觉与语言表征。我们提取关键点与提议点来建模3D上下文与实例，并提出带上下文调制的点-语言对齐（point-language alignment with context modulation，PLACM）机制，该机制学习将词级和句级语言嵌入与视觉表征逐步对齐，同时通过视觉上下文的调制捕捉潜在的信息关联。为进一步捕捉全局与局部关系，我们提出空间多粒度建模方案，将PLACM同时应用于全局和局部场域。实验结果证明了HAM的优越性，可视化结果表明它能够动态建模细粒度的视觉与语言表征。HAM大幅优于现有方法，在两个公开数据集上取得了当前最优性能，并在ECCV 2022 ScanRefer挑战赛中获得冠军。代码可在https://github.com/PPjmchen/HAM 获取。
英文摘要：
This paper presents a novel hierarchical alignment model (HAM) that learns multi-granularity visual and linguistic representations in an end-to-end manner. We extract key points and proposal points to model 3D contexts and instances, and propose point-language alignment with context modulation (PLACM) mechanism, which learns to gradually align word-level and sentence-level linguistic embeddings with visual representations, while the modulation with the visual context captures latent informative relationships. To further capture both global and local relationships, we propose a spatially multi-granular modeling scheme that applies PLACM to both global and local fields. Experimental results demonstrate the superiority of HAM, with visualized results showing that it can dynamically model fine-grained visual and linguistic representations. HAM outperforms existing methods by a significant margin and achieves state-of-the-art performance on two publicly available datasets, and won the championship in ECCV 2022 ScanRefer challenge. Code is available at~\url{https://github.com/PPjmchen/HAM}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
