面向广义指代表达理解的层次对齐增强自适应定位网络
Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension
AI总结：
本文提出层次对齐增强自适应定位网络HieA2G，通过层次多模态语义对齐和自适应定位计数器解决广义指代表达理解中无目标、单目标与多目标情况，并在GREC、REC、RES等多个任务上取得最优性能。
摘要：
本文针对广义指代表达理解（GREC）这一具有挑战性的任务展开研究。与专注于单目标指代表达的经典指代表达理解（REC）相比，GREC进一步涵盖无目标和多目标指代表达，将任务扩展到更贴近实际应用的场景。现有REC方法在处理GREC中遇到的复杂情况时面临困难，主要原因在于其固定输出形式以及多模态表示能力的局限。为解决这些问题，我们提出了一种面向GREC的层次对齐增强自适应定位网络（HieA2G），该网络能够灵活处理各种类型的指代表达。首先，我们提出层次多模态语义对齐（HMSA）模块，融合词-物体、短语-物体以及文本-图像三个层次的对齐。该模块能够在多个层级上实现层次化跨模态交互，从而获得全面且鲁棒的多模态理解，显著增强复杂情况下的定位能力。其次，为应对GREC中目标物体数量变化的问题，我们引入自适应定位计数器（AGC），用于动态确定输出目标的数量。此外，在AGC中采用辅助对比损失，通过拉近具有相同计数的多模态特征、推开具有不同计数的多模态特征来增强物体计数能力。大量实验结果表明，HieA2G在具有挑战性的GREC任务以及其他4个任务上均取得了新的最优性能，这些任务包括REC、短语定位、指代表达分割（RES）和广义指代表达分割（GRES），充分证明了所提出的HieA2G具有显著的优越性和泛化能力。
英文摘要：
In this work, we address the challenging task of Generalized Referring Expression Comprehension (GREC). Compared to the classic Referring Expression Comprehension (REC) that focuses on single-target expressions, GREC extends the scope to a more practical setting by further encompassing no-target and multi-target expressions. Existing REC methods face challenges in handling the complex cases encountered in GREC, primarily due to their fixed output and limitations in multi-modal representations. To address these issues, we propose a Hierarchical Alignment-enhanced Adaptive Grounding Network (HieA2G) for GREC, which can flexibly deal with various types of referring expressions. First, a Hierarchical Multi-modal Semantic Alignment (HMSA) module is proposed to incorporate three levels of alignments, including word-object, phrase-object, and text-image alignment. It enables hierarchical cross-modal interactions across multiple levels to achieve comprehensive and robust multi-modal understanding, greatly enhancing grounding ability for complex cases. Then, to address the varying number of target objects in GREC, we introduce an Adaptive Grounding Counter (AGC) to dynamically determine the number of output targets. Additionally, an auxiliary contrastive loss is employed in AGC to enhance object-counting ability by pulling in multi-modal features with the same counting and pushing away those with different counting. Extensive experimental results show that HieA2G achieves new state-of-the-art performance on the challenging GREC task and also the other 4 tasks, including REC, Phrase Grounding, Referring Expression Segmentation (RES), and Generalized Referring Expression Segmentation (GRES), demonstrating the remarkable superiority and generalizability of the proposed HieA2G.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
