用于场景图生成增强的视觉-语言模型集成中的谓词去偏
Predicate Debiasing in Vision-Language Models Integration for Scene Graph Generation Enhancement
AI总结：
本文提出集成预训练视觉-语言模型增强场景图生成表示，通过LM估计近似不可获取的谓词分布以缓解预训练偏差，并设计确定性感知指标动态调整集成权重，免训练显著提升性能。
摘要：
场景图生成（SGG）提供视觉场景的基础语言表示，要求模型掌握对象间复杂多样的语义。SGG中的复杂性和多样性导致表示不足，部分三元组标签在训练期间罕见甚至未见，导致预测不精确。为此，我们提出集成预训练视觉-语言模型以增强表示。然而，由于预训练与SGG之间的差距，直接在SGG上推理预训练VLMs会导致严重偏差，这源于预训练语言集中谓词分布的不平衡。为缓解该偏差，我们引入一种新颖的LM估计来近似无法获取的谓词分布。最后，我们将去偏后的VLMs与SGG模型集成以增强表示，其中我们设计了一个确定性感知指标来对每个样本评分并动态调整集成权重。我们的免训练方法有效解决了预训练VLMs中的谓词偏差，增强了SGG的表示，并显著提升了性能。
英文摘要：
Scene Graph Generation (SGG) provides basic language representation of visual scenes, requiring models to grasp complex and diverse semantics between objects. This complexity and diversity in SGG leads to underrepresentation, where parts of triplet labels are rare or even unseen during training, resulting in imprecise predictions. To tackle this, we propose integrating the pretrained Vision-language Models to enhance representation. However, due to the gap between pretraining and SGG, direct inference of pretrained VLMs on SGG leads to severe bias, which stems from the imbalanced predicates distribution in the pretraining language set. To alleviate the bias, we introduce a novel LM Estimation to approximate the unattainable predicates distribution. Finally, we ensemble the debiased VLMs with SGG models to enhance the representation, where we design a certainty-aware indicator to score each sample and dynamically adjust the ensemble weights. Our training-free method effectively addresses the predicates bias in pretrained VLMs, enhances SGG's representation, and significantly improve the performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
