DeiSAM:基于指示性提示的 Segment Anything
DeiSAM: Segment Anything with Deictic Prompting
AI总结:
本文提出 DeiSAM,将 LLM 的一阶逻辑规则生成、场景图可微推理与预训练分割模型结合,并构建 DeiVG 数据集,显著提升复杂指示性语言提示下的图像分割效果。
摘要:
大规模预训练神经网络已在包括零样本图像分割在内的多种任务中展现出强大能力。为了识别复杂场景中的具体对象,人类会本能地依赖自然语言中的指示性描述,即根据语境指称某物,例如“在桌子上且位于杯子后面的那个物体”。然而,由于深度学习方法缺乏在复杂场景中的推理能力,它们无法可靠地解释此类指示性表达。为解决这一问题,我们提出 DeiSAM——一种将大型预训练神经网络与可微逻辑推理器相结合的方法,用于指示性可提示分割。给定复杂的文本分割描述,DeiSAM 利用大型语言模型(LLM)生成一阶逻辑规则,并在生成的场景图上执行可微前向推理。随后,DeiSAM 通过将对象与逻辑推断出的图像区域进行匹配来分割对象。作为评估的一部分,我们提出了 Deictic Visual Genome(DeiVG)数据集,其中包含成对的视觉输入与复杂的指示性文本提示。实验结果表明,在指示性可提示分割任务上,DeiSAM 相比纯数据驱动的基线取得了显著提升。
英文摘要:
Large-scale, pre-trained neural networks have demonstrated strong capabilities in various tasks, including zero-shot image segmentation. To identify concrete objects in complex scenes, humans instinctively rely on deictic descriptions in natural language, i.e., referring to something depending on the context such as "The object that is on the desk and behind the cup.". However, deep learning approaches cannot reliably interpret such deictic representations due to their lack of reasoning capabilities in complex scenarios. To remedy this issue, we propose DeiSAM -- a combination of large pre-trained neural networks with differentiable logic reasoners -- for deictic promptable segmentation. Given a complex, textual segmentation description, DeiSAM leverages Large Language Models (LLMs) to generate first-order logic rules and performs differentiable forward reasoning on generated scene graphs. Subsequently, DeiSAM segments objects by matching them to the logically inferred image regions. As part of our evaluation, we propose the Deictic Visual Genome (DeiVG) dataset, containing paired visual input and complex, deictic textual prompts. Our empirical results demonstrate that DeiSAM is a substantial improvement over purely data-driven baselines for deictic promptable segmentation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。