如何理解“Support”？一种用于弱监督短语定位的隐式增强因果推断方法
How to Understand "Support"? An Implicit-enhanced Causal Inference Approach for Weakly-supervised Phrase Grounding
AI总结：
针对弱监督短语定位中被忽视的隐式短语-区域关系，本文提出结合干预与反事实的IECI因果推断方法，并构建隐式增强数据集，验证其显著优于现有基线及多模态大语言模型。
摘要：
弱监督短语定位（Weakly-supervised Phrase Grounding, WPG）是一项新兴任务，旨在推断细粒度的短语-区域匹配关系，而训练时仅利用粗粒度的句子-图像对。然而，现有WPG研究在很大程度上忽略了隐式短语-区域匹配关系；这些关系对于评估模型理解深层多模态语义的能力至关重要。为此，本文提出一种隐式增强因果推断（Implicit-Enhanced Causal Inference, IECI）方法，以应对建模隐式关系并在显式关系之外突出隐式关系的挑战。具体而言，该方法分别利用干预和反事实技术来解决上述两个挑战。此外，本文标注了一个高质量的隐式增强数据集以评估IECI，详细评估表明IECI相对于最先进基线具有显著优势。特别地，我们观察到一个有趣的发现：在该隐式增强数据集上，IECI大幅优于先进的多模态大语言模型，这可能促进更多研究在此方向上评估多模态大语言模型。
英文摘要：
Weakly-supervised Phrase Grounding (WPG) is an emerging task of inferring the fine-grained phrase-region matching, while merely leveraging the coarse-grained sentence-image pairs for training. However, existing studies on WPG largely ignore the implicit phrase-region matching relations, which are crucial for evaluating the capability of models in understanding the deep multimodal semantics. To this end, this paper proposes an Implicit-Enhanced Causal Inference (IECI) approach to address the challenges of modeling the implicit relations and highlighting them beyond the explicit. Specifically, this approach leverages both the intervention and counterfactual techniques to tackle the above two challenges respectively. Furthermore, a high-quality implicit-enhanced dataset is annotated to evaluate IECI and detailed evaluations show the great advantages of IECI over the state-of-the-art baselines. Particularly, we observe an interesting finding that IECI outperforms the advanced multimodal LLMs by a large margin on this implicit-enhanced dataset, which may facilitate more research to evaluate the multimodal LLMs in this direction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
