VLPrompt：面向全景场景图生成的视觉-语言提示方法
VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generation
AI总结：
针对全景场景图生成中关系长尾问题，提出VLPrompt模型，结合图像视觉信息与大语言模型的语言信息，通过注意力提示网络精准预测关系，在PSG数据集上超越现有最优方法。
摘要：
全景场景图生成（Panoptic Scene Graph Generation, PSG）旨在通过同时分割物体并预测物体间的关系，实现全面的图像理解。然而，关系中的长尾问题导致其在实际应用中的效果不尽如人意。现有方法主要依赖视觉信息，或仅利用有限的语言信息（如物体或关系名称），从而忽略了语言信息的效用。借助大语言模型（Large Language Models, LLMs）的最新进展，我们提出利用语言信息辅助关系预测，尤其是针对稀有关系的预测。为此，我们提出视觉-语言提示（Vision-Language Prompting, VLPrompt）模型，该模型从图像中获取视觉信息，从LLMs中获取语言信息，随后通过基于注意力机制的提示网络实现精准的关系预测。我们的大量实验表明，VLPrompt在PSG数据集上的表现显著优于此前的最优方法，证明了引入语言信息的有效性，且能缓解关系的长尾问题。代码可在https://github.com/franciszzj/TP-SIS获取。
英文摘要：
Panoptic Scene Graph Generation (PSG) aims at achieving a comprehensive image understanding by simultaneously segmenting objects and predicting relations among objects. However, the long-tail problem among relations leads to unsatisfactory results in real-world applications. Prior methods predominantly rely on vision information or utilize limited language information, such as object or relation names, thereby overlooking the utility of language information. Leveraging the recent progress in Large Language Models (LLMs), we propose to use language information to assist relation prediction, particularly for rare relations. To this end, we propose the Vision-Language Prompting (VLPrompt) model, which acquires vision information from images and language information from LLMs. Then, through a prompter network based on attention mechanism, it achieves precise relation prediction. Our extensive experiments show that VLPrompt significantly outperforms previous state-of-the-art methods on the PSG dataset, proving the effectiveness of incorporating language information and alleviating the long-tail problem of relations. Code is available at \url{https://github.com/franciszzj/TP-SIS}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
