基于生成式场景图构建的生成式视觉常识问答与解释
Generative Visual Commonsense Answering and Explaining with Generative Scene Graph Constructing
AI总结：
针对现有视觉常识推理过度依赖训练记忆、未有效利用场景对象关系的问题，提出场景图增强的G2方法，通过构建无位置场景图并结合过滤选择策略实现问答与解释，实验验证了框架有效性。
摘要：
视觉常识推理（Visual Commonsense Reasoning）被视为实现高级视觉场景理解的一项挑战性任务，已被用于评估AI系统的推理能力。然而，可靠的推理需要充分掌握场景细节。现有研究未能有效利用场景中存在的现实世界对象关系信息，反而过度依赖训练记忆中的知识。基于这些观察，我们提出了一种新颖的场景图增强视觉常识推理生成方法，名为\textit{\textbf{G2}}，该方法首先利用图像块和大语言模型（LLMs）构建无位置场景图，然后基于场景图信息进行问答与解释。我们还提出了自动场景图过滤与选择策略，以在训练过程中吸收有价值的场景图信息。我们分别在场景图构建、视觉常识问答与解释的任务和数据集上开展了大量实验。实验结果与消融分析证明了我们所提出框架的有效性。
英文摘要：
Visual Commonsense Reasoning, which is regarded as one challenging task to pursue advanced visual scene comprehension, has been used to diagnose the reasoning ability of AI systems. However, reliable reasoning requires a good grasp of the scene's details. Existing work fails to effectively exploit the real-world object relationship information present within the scene, and instead overly relies on knowledge from training memory. Based on these observations, we propose a novel scene-graph-enhanced visual commonsense reasoning generation method named \textit{\textbf{G2}}, which first utilizes the image patches and LLMs to construct a location-free scene graph, and then answer and explain based on the scene graph's information. We also propose automatic scene graph filtering and selection strategies to absorb valuable scene graph information during training. Extensive experiments are conducted on the tasks and datasets of scene graph constructing and visual commonsense answering and explaining, respectively. Experimental results and ablation analysis demonstrate the effectiveness of our proposed framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
