arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.08305cs.CV
收藏

迈向灵活的视觉关系分割

Towards Flexible Visual Relationship Segmentation

Northeastern University(东北大学);Microsoft Research(微软研究院)

Fangrui Zhu, Jianwei Yang, Huaizu Jiang

AI总结:

针对视觉关系理解相关任务分散研究的问题,提出整合HOI检测、SGG、RR任务且支持开放词汇的FleVRS模型,利用多模态协同实现关系定位,在多数据集上性能优于现有模型。

摘要:

视觉关系理解已在人-物交互(HOI)检测、场景图生成(SGG)和指代关系(RR)任务中被分别研究。鉴于这些任务的复杂性和相互关联性,拥有一个能以连贯方式有效解决这些任务的灵活框架至关重要。在本研究中,我们提出FleVRS,这一单一模型可在标准和可提示视觉关系分割中无缝整合上述三个方面,还具备开放词汇分割能力以适应新场景。FleVRS利用文本与图像模态之间的协同作用,从图像中定位各类关系,并借助视觉-语言模型的文本特征实现视觉概念理解。在多个数据集上的实证验证表明,我们的框架在标准、可提示和开放词汇任务中均优于现有模型,例如在HICO-DET上mAP提升1.9,在VRD上Acc提升11.4,在未见过的HICO-DET上mAP提升4.7。我们的FleVRS朝着更直观、全面且可扩展的视觉关系理解迈出了重要一步。

英文摘要:

Visual relationship understanding has been studied separately in human-object interaction(HOI) detection, scene graph generation(SGG), and referring relationships(RR) tasks. Given the complexity and interconnectedness of these tasks, it is crucial to have a flexible framework that can effectively address these tasks in a cohesive manner. In this work, we propose FleVRS, a single model that seamlessly integrates the above three aspects in standard and promptable visual relationship segmentation, and further possesses the capability for open-vocabulary segmentation to adapt to novel scenarios. FleVRS leverages the synergy between text and image modalities, to ground various types of relationships from images and use textual features from vision-language models to visual conceptual understanding. Empirical validation across various datasets demonstrates that our framework outperforms existing models in standard, promptable, and open-vocabulary tasks, e.g., +1.9 $mAP$ on HICO-DET, +11.4 $Acc$ on VRD, +4.7 $mAP$ on unseen HICO-DET. Our FleVRS represents a significant step towards a more intuitive, comprehensive, and scalable understanding of visual relationships.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑