面向开放词汇场景图生成的关系感知分层提示
Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation
AI总结：
针对开放词汇场景图生成中文本表示固定的问题，提出关系感知分层提示框架，通过整合主客体信息、利用LLM生成区域感知提示并引入动态选择机制，在多个数据集上实现了最先进的性能。
摘要：
开放词汇场景图生成（OV-SGG）通过将视觉关系表示与开放词汇文本表示对齐，克服了闭集假设的限制。这使得识别新颖视觉关系成为可能，从而适用于具有多样关系的现实场景。然而，现有的OV-SGG方法受限于固定的文本表示，限制了图文对齐的多样性和准确性。为了解决这些挑战，我们提出了关系感知分层提示（RAHP）框架，通过整合主客体和特定区域的关系信息来增强文本表示。我们的方法利用实体聚类来处理关系三元组类别的复杂性，从而实现主客体信息的有效整合。此外，我们利用大型语言模型（LLM）生成详细的区域感知提示，捕获细粒度的视觉交互，并改善视觉与文本模态之间的对齐。RAHP还在视觉语言模型（VLM）中引入了动态选择机制，该机制根据视觉内容自适应地选择相关文本提示，从而减少无关提示产生的噪声。在Visual Genome和Open Images v6数据集上的大量实验表明，我们的框架持续取得最先进的性能，证明了其在解决开放词汇场景图生成挑战方面的有效性。代码可在以下地址获取：https://github.com/Leon022/RAHP
英文摘要：
Open-vocabulary Scene Graph Generation (OV-SGG) overcomes the limitations of the closed-set assumption by aligning visual relationship representations with open-vocabulary textual representations. This enables the identification of novel visual relationships, making it applicable to real-world scenarios with diverse relationships. However, existing OV-SGG methods are constrained by fixed text representations, limiting diversity and accuracy in image-text alignment. To address these challenges, we propose the Relation-Aware Hierarchical Prompting (RAHP) framework, which enhances text representation by integrating subject-object and region-specific relation information. Our approach utilizes entity clustering to address the complexity of relation triplet categories, enabling the effective integration of subject-object information. Additionally, we utilize a large language model (LLM) to generate detailed region-aware prompts, capturing fine-grained visual interactions and improving alignment between visual and textual modalities. RAHP also introduces a dynamic selection mechanism within Vision-Language Models (VLMs), which adaptively selects relevant text prompts based on the visual content, reducing noise from irrelevant prompts. Extensive experiments on the Visual Genome and Open Images v6 datasets demonstrate that our framework consistently achieves state-of-the-art performance, demonstrating its effectiveness in addressing the challenges of open-vocabulary scene graph generation. The code is available at: https://github.com/Leon022/RAHP
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
