人类天生具备选择性视觉感知能力,能够将视野中的特定区域转化为结构化符号知识,并根据目标分配有限的注意力资源,推理不同区域之间的关系。尽管这对人类而言稀松平常,但由于所需的认知能力和常识知识极为复杂,当前的感知系统在提取结构化信息方面仍存在不足。为填补这一空白,我们提出了一项名为“局部场景图生成”的新任务。与传统场景图生成任务需要生成图像中所有对象及其关系不同,本任务旨在提取包含部分对象及其关系的相关结构化信息,以助力需要高级理解与推理能力的下游任务。相应地,我们提出了zEro-shot Local scEne GrAph geNeraTion(ELEGANT,零样本局部场景图生成)框架,该框架利用以强大感知和常识推理能力著称的基础模型,通过基础模型之间的协作与信息交互取得更优效果,无需标注监督即可实现零样本局部场景图生成。此外,我们提出了一种新颖的开放式评估指标Entity-level CLIPScorE(ECLIPSE,实体级CLIP得分),它突破了以往闭集评估指标有限的标签空间限制,能够提供更全面的评估。实验结果表明,我们的方法在开放式评估设置下显著优于基线模型,在闭集设置下也比现有方法实现了最高达24.58%的性能提升,证明了所提框架的有效性和强大的推理能力。
英文摘要:
Humans inherently recognize objects via selective visual perception, transform specific regions from the visual field into structured symbolic knowledge, and reason their relationships among regions based on the allocation of limited attention resources in line with humans' goals. While it is intuitive for humans, contemporary perception systems falter in extracting structural information due to the intricate cognitive abilities and commonsense knowledge required. To fill this gap, we present a new task called Local Scene Graph Generation. Distinct from the conventional scene graph generation task, which encompasses generating all objects and relationships in an image, our proposed task aims to abstract pertinent structural information with partial objects and their relationships for boosting downstream tasks that demand advanced comprehension and reasoning capabilities. Correspondingly, we introduce zEro-shot Local scEne GrAph geNeraTion (ELEGANT), a framework harnessing foundation models renowned for their powerful perception and commonsense reasoning, where collaboration and information communication among foundation models yield superior outcomes and realize zero-shot local scene graph generation without requiring labeled supervision. Furthermore, we propose a novel open-ended evaluation metric, Entity-level CLIPScorE (ECLIPSE), surpassing previous closed-set evaluation metrics by transcending their limited label space, offering a broader assessment. Experiment results show that our approach markedly outperforms baselines in the open-ended evaluation setting, and it also achieves a significant performance boost of up to 24.58% over prior methods in the close-set setting, demonstrating the effectiveness and powerful reasoning ability of our proposed framework.
Vision-Language Integration for Zero-Shot Scene Understanding in Real-World Environments
Nitte (Deemed to Be University), Nitte Institute of Professional Education(尼特大学(视为大学),尼特专业教育学院);Ballari Institute of Technology and Management(巴拉里技术与管理学院)
One-Shot Crowd Counting With Density Guidance For Scene Adaptation
Jiangsu Key Laboratory of Intelligent Weather Forecasting and Applications Based on Big Data(江苏大数据智能天气预报与应用重点实验室);State Key Laboratory of Climate System Prediction and Risk Management (CPRM)(气候变化预测与风险管理国家重点实验室);ICAR/CIC-FEMD/KLME/ILCEC;Nanjing University of Information Science and Technology(南京信息工程大学);School of Artificial Intelligence, OPtics and ElectroNics, Northwestern Polytechnical University(人工智能、光学与电子学学院,西北工业大学);School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院)
Feature Transformation for Cross-domain Few-shot Remote Sensing Scene Classification
South China Agricultural University(华南农业大学);Pazhou Lab(琶洲实验室)
由于遥感图像空间分辨率的不断提高和遥感图像间差异的不断增大,遥感场景的有效分类仍然是一个挑战。现有的研究已经大大提高了遥感场景分类的性能。然而,这些方法不适用于跨域Few-Shot问题,其中目标域具有非常有限的训练样本,并且具有与源域不同的数据分布。为了提高模型的适用性,本文提出了特征变换模块(FTM)。FTM通过一个非常简单的仿射操作将源域上学习到的特征分布转移到目标域上,而附加参数可以忽略不计。此外,FTM可以有效地学习目标域的情况下,很少的训练数据可用,是不可知的特定的网络结构。RSSC和土地覆盖制图任务的实验验证了其处理跨域Few-Shot问题的能力。与直接微调相比,FTM具有更好的性能,并具有更好的可移植性和细粒度的区分力。\textit{Code will be publicly available.}
University of Electronic Science and Technology of China(电子科技大学)
视觉场景图(视觉关系)作为图像内容的结构化表示,在计算机视觉和自然语言处理之间起着桥梁作用。众所周知,场景图生成任务中的现有模型需要数十或数百个标记样本。相比之下,人类可以从几个甚至一个例子中学习视觉关系。受此启发,我们设计了一个名为One-Shot Scene Graph Generation的任务,其中每个关系三元组(例如,“dog-has-head”)仅来自一个标记的示例。关键的见解是,而不是从头开始学习,一个人可以利用丰富的先验知识。在本文中,我们提出了多重结构化知识(关系知识和常识知识)的一杆场景图生成任务。具体地,关系知识表示从视觉内容提取的实体之间的关系的先验知识,例如,“狗”与“院子”之间可能存在“站在”、“坐在”、“躺在”的视觉关系,而常识知识则编码了“狗能守院子”等“造义”知识。通过将这两种知识组织成图结构,图卷积网络(GCN)被用来提取实体的知识嵌入语义特征。此外,我们不是从Faster R-CNN生成的每个实体中提取孤立的视觉特征,而是利用实例关系Transformer编码器来充分探索它们的上下文信息。基于构造的单次数据集,实验结果表明,我们的方法显着优于现有的国家的最先进的方法,由一个大的利润。消融研究也验证了实例关系Transformer编码器和多结构化知识的有效性。
Prompt-guided Scene Generation for 3D Zero-Shot Learning
The University of New South Wales(新南威尔士大学);Australian National University(澳大利亚国立大学);North South University(北南大学);University of Technology Sydney(悉尼科技大学);Commonwealth Scientific and Industrial Research Organisation(联邦科学与工业研究组织)
Deep Complementary Joint Model for Complex Scene Registration and Few-shot Segmentation on Medical Images
Southeast University(东南大学);Centre de Recherche en Information Biomédicale Sino-Français (CRIBs)(中法生物医学信息研究中心);Univ Rennes(雷恩大学);Inserm(法国国家健康与医学研究院);LTSI - UMR1099(信号与图像处理实验室)
CycleSAM: Few-Shot Surgical Scene Segmentation with Cycle- and Scene-Consistent Feature Matching
University of Strasbourg, CNRS, INSERM, ICube, UMR7357(斯特拉斯堡大学、国家科学研究中心、法国国家医学研究院、ICube、UMR7357);IHU Strasbourg(斯特拉斯堡IHU);Fondazione Policlinico Universitario A. Gemelli IRCCS(阿·吉梅利大学附属医院基金会)
The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning
School of Computer Science and Technology, University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院);Macquarie University(麦考瑞大学);Australian Institute for Machine Learning, The University of Adelaide(阿德莱德大学澳大利亚机器学习研究所)
ZeroPose: CAD-Prompted Zero-shot Object 6D Pose Estimation in Cluttered Scenes
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳)计算机科学与技术学院);Pengcheng Laboratory(鹏城实验室);SenseTime Research(商汤科技研究院)