LLM4SGG：用于弱监督场景图生成的大型语言模型
LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation
AI总结：
提出LLM4SGG方法，利用大型语言模型结合思维链和少样本学习，解决弱监督场景图生成中三元组提取的语义过度简化和对齐时的低密度图问题，在多个数据集上显著提升了性能且具备数据效率。
摘要：
弱监督场景图生成（WSSGG）研究近期作为严重依赖昂贵标注的完全监督方法的替代方案而出现。在此方面，WSSGG的研究利用图像描述来获取未定位的三元组，同时主要关注将未定位的三元组定位到图像区域上。然而，它们忽略了从描述生成三元组的过程中涉及的两个问题：1）从描述中提取三元组时会出现语义过度简化问题，描述中的细粒度谓词被不合意地转换为粗粒度谓词，导致长尾谓词分布；2）将描述中的三元组与感兴趣的实体/谓词类对齐时会出现低密度场景图问题，许多三元组被丢弃且未在训练中使用，导致监督不足。为了解决这两个问题，我们提出了一种新方法，即用于弱监督SGG的大型语言模型（LLM4SGG），我们通过在从描述中提取三元组以及将实体/谓词类与目标数据对齐期间利用LLM对语言的深入理解和推理能力来缓解这两个问题。为了进一步让LLM参与这些过程，我们采用了思维链和上下文少样本学习策略的思想。为了验证LLM4SGG的有效性，我们在Visual Genome和GQA数据集上进行了广泛实验，与最先进的WSSGG方法相比，在Recall@K和mean Recall@K上均显示出显著改进。LLM4SGG的另一个吸引人之处是它具有数据效率，能够使用少量训练图像进行有效的模型训练。
英文摘要：
Weakly-Supervised Scene Graph Generation (WSSGG) research has recently emerged as an alternative to the fully-supervised approach that heavily relies on costly annotations. In this regard, studies on WSSGG have utilized image captions to obtain unlocalized triplets while primarily focusing on grounding the unlocalized triplets over image regions. However, they have overlooked the two issues involved in the triplet formation process from the captions: 1) Semantic over-simplification issue arises when extracting triplets from captions, where fine-grained predicates in captions are undesirably converted into coarse-grained predicates, resulting in a long-tailed predicate distribution, and 2) Low-density scene graph issue arises when aligning the triplets in the caption with entity/predicate classes of interest, where many triplets are discarded and not used in training, leading to insufficient supervision. To tackle the two issues, we propose a new approach, i.e., Large Language Model for weakly-supervised SGG (LLM4SGG), where we mitigate the two issues by leveraging the LLM's in-depth understanding of language and reasoning ability during the extraction of triplets from captions and alignment of entity/predicate classes with target data. To further engage the LLM in these processes, we adopt the idea of Chain-of-Thought and the in-context few-shot learning strategy. To validate the effectiveness of LLM4SGG, we conduct extensive experiments on Visual Genome and GQA datasets, showing significant improvements in both Recall@K and mean Recall@K compared to the state-of-the-art WSSGG methods. A further appeal is that LLM4SGG is data-efficient, enabling effective model training with a small amount of training images.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
