arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.01356cs.CV
收藏

少即是多:基于基础模型的零样本局部场景图生成

Less is More: Toward Zero-Shot Local Scene Graph Generation via Foundation Models

Pennsylvania State University(宾夕法尼亚州立大学)

Shu Zhao, Huijuan Xu

AI总结:

该研究提出局部场景图生成新任务,构建基于多基础模型协作的零样本ELEGANT框架,还设计了开放式ECLIPSE评估指标,在开/闭集设置下均取得性能提升。

摘要:

人类天生具备选择性视觉感知能力,能够将视野中的特定区域转化为结构化符号知识,并根据目标分配有限的注意力资源,推理不同区域之间的关系。尽管这对人类而言稀松平常,但由于所需的认知能力和常识知识极为复杂,当前的感知系统在提取结构化信息方面仍存在不足。为填补这一空白,我们提出了一项名为“局部场景图生成”的新任务。与传统场景图生成任务需要生成图像中所有对象及其关系不同,本任务旨在提取包含部分对象及其关系的相关结构化信息,以助力需要高级理解与推理能力的下游任务。相应地,我们提出了zEro-shot Local scEne GrAph geNeraTion(ELEGANT,零样本局部场景图生成)框架,该框架利用以强大感知和常识推理能力著称的基础模型,通过基础模型之间的协作与信息交互取得更优效果,无需标注监督即可实现零样本局部场景图生成。此外,我们提出了一种新颖的开放式评估指标Entity-level CLIPScorE(ECLIPSE,实体级CLIP得分),它突破了以往闭集评估指标有限的标签空间限制,能够提供更全面的评估。实验结果表明,我们的方法在开放式评估设置下显著优于基线模型,在闭集设置下也比现有方法实现了最高达24.58%的性能提升,证明了所提框架的有效性和强大的推理能力。

英文摘要:

Humans inherently recognize objects via selective visual perception, transform specific regions from the visual field into structured symbolic knowledge, and reason their relationships among regions based on the allocation of limited attention resources in line with humans' goals. While it is intuitive for humans, contemporary perception systems falter in extracting structural information due to the intricate cognitive abilities and commonsense knowledge required. To fill this gap, we present a new task called Local Scene Graph Generation. Distinct from the conventional scene graph generation task, which encompasses generating all objects and relationships in an image, our proposed task aims to abstract pertinent structural information with partial objects and their relationships for boosting downstream tasks that demand advanced comprehension and reasoning capabilities. Correspondingly, we introduce zEro-shot Local scEne GrAph geNeraTion (ELEGANT), a framework harnessing foundation models renowned for their powerful perception and commonsense reasoning, where collaboration and information communication among foundation models yield superior outcomes and realize zero-shot local scene graph generation without requiring labeled supervision. Furthermore, we propose a novel open-ended evaluation metric, Entity-level CLIPScorE (ECLIPSE), surpassing previous closed-set evaluation metrics by transcending their limited label space, offering a broader assessment. Experiment results show that our approach markedly outperforms baselines in the open-ended evaluation setting, and it also achieves a significant performance boost of up to 24.58% over prior methods in the close-set setting, demonstrating the effectiveness and powerful reasoning ability of our proposed framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑