There is a surge of interest in image scene graph generation (object, attribute and relationship detection) due to the need of building fine-grained image understanding models that go beyond object detection. Due to the lack of a good benchmark, the reported results of different scene graph generation models are not directly comparable, impeding the research progress. We have developed a much-needed scene graph generation benchmark based on the maskrcnn-benchmark and several popular models. This paper presents main features of our benchmark and a comprehensive ablation study of scene graph generation models using the Visual Genome and OpenImages Visual relationship detection datasets. Our codebase is made publicly available at https://github.com/microsoft/scene_graph_benchmark.
The Scene Graph Generation (SGG) task aims to detect all the objects and their pairwise visual relationships in a given image. Although SGG has achieved remarkable progress over the last few years, almost all existing SGG models follow the same training paradigm: they treat both object and predicate classification in SGG as a single-label classification problem, and the ground-truths are one-hot target labels. However, this prevalent training paradigm has overlooked two characteristics of current SGG datasets: 1) For positive samples, some specific subject-object instances may have multiple re…
Adaptive Visual Scene Understanding: Incremental Scene Graph Generation
College of Computing and Data Science, Nanyang Technological University (NTU)（南洋理工大学计算与数据科学学院）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）
NICEST: Noisy Label Correction and Training for Robust Scene Graph Generation
College of Computer Science, Zhejiang University（浙江大学计算机学院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Data Platform, Tencent（腾讯数据平台）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）
Modeling Scientific Experiment Scenes: Dataset and Model
College of Computer Science and Engineering, Shandong University of Science and Technology（山东科技大学计算机科学与工程学院）；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；ABC FINTECH Company Limited（ABC金融科技有限公司）；NOVA Information Management School, Universidade Nova de Lisboa（里斯本新大学NOVA信息管理学院）；School of Biomedical Engineering, Shenzhen University（深圳大学生物医学工程学院）
National University of Singapore（新加坡国立大学）；Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
Scene graph generation (SGG) is designed to extract (subject, predicate, object) triplets in images. Recent works have made a steady progress on SGG, and provide useful tools for high-level vision and language understanding. However, due to the data distribution problems including long-tail distribution and semantic ambiguity, the predictions of current SGG models tend to collapse to several frequent but uninformative predicates (e.g., on, at), which limits practical application of these models in downstream tasks. To deal with the problems above, we propose a novel Internal and External Data …
Relation prediction among entities in images is an important step in scene graph generation (SGG), which further impacts various visual understanding and reasoning tasks. Existing SGG frameworks, however, require heavy training yet are incapable of modeling unseen (i.e.,zero-shot) triplets. In this work, we stress that such incapability is due to the lack of commonsense reasoning,i.e., the ability to associate similar entities and infer similar relations based on general understanding of the world. To fill this gap, we propose CommOnsense-integrAted sCenegrapHrElation pRediction (COACHER), a f…
STAR: A First-Ever Dataset and A Large-Scale Benchmark for Scene Graph Generation in Large-Size Satellite Imagery
School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）；OpenGVLab, Shanghai AI Laboratory（上海人工智能实验室OpenGVLab）；School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University（西北工业大学人工智能、光学与电子学院（iOPEN））；Department of Computer Science, Cornell University（康奈尔大学计算机科学系）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Geosciences and Info-Physics, Central South University（中南大学地球科学与信息物理学院）；School of Automation, Southeast University（东南大学自动化学院）；School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）
Scene graph generation (SGG) is to detect object pairs with their relations in an image. Existing SGG approaches often use multi-stage pipelines to decompose this task into object detection, relation graph construction, and dense or dense-to-sparse relation prediction. Instead, from a perspective on SGG as a direct set prediction, this paper presents a simple, sparse, and unified framework, termed as Structured Sparse R-CNN. The key to our method is a set of learnable triplet queries and a structured triplet detector which could be jointly optimized from the training set in an end-to-end manne…
For a typical Scene Graph Generation (SGG) method, there is often a large gap in the performance of the predicates' head classes and tail classes. This phenomenon is mainly caused by the semantic overlap between different predicates as well as the long-tailed data distribution. In this paper, a Predicate Correlation Learning (PCL) method for SGG is proposed to address the above two problems by taking the correlation between predicates into consideration. To describe the semantic overlap between strong-correlated predicate classes, a Predicate Correlation Matrix (PCM) is defined to quantify the…
Revisiting Scene Graph Generation from the Perspective of Detector-Conditioned Reachability
Technische Universität Berlin（柏林工业大学）；Humboldt Universität zu Berlin（柏林洪堡大学）；Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK（格勒诺布尔大学、法国国家信息与自动化研究所、法国国家科学研究中心、格勒诺布尔国立综合理工学院、法国国家科学研究中心格勒诺布尔计算机科学实验室）；Bernstein Center for Computational Neuroscience（伯恩斯坦计算神经科学中心）；Science of Intelligence Research Cluster of Excellence（智能科学卓越研究集群）
Zhejiang University（浙江大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Zhejiang University Embedded System Engineering Research Center, Ministry of Education of China（浙江大学教育部嵌入式系统工程研究中心）；Zhejiang Provincial Key Laboratory for Network Multimedia Technologies（浙江省网络多媒体技术重点实验室）
Today, scene graph generation(SGG) task is largely limited in realistic scenarios, mainly due to the extremely long-tailed bias of predicate annotation distribution. Thus, tackling the class imbalance trouble of SGG is critical and challenging. In this paper, we first discover that when predicate labels have strong correlation with each other, prevalent re-balancing strategies(e.g., re-sampling and re-weighting) will give rise to either over-fitting the tail data(e.g., bench sitting on sidewalk rather than on), or still suffering the adverse effect from the original uneven distribution(e.g., a…
Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention
The Hong Kong Polytechnic University（香港理工大学）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（香港中文大学（深圳）人工智能与机器人中心（HKISI-CAS））；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所 多模态人工智能系统国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学 人工智能学院）
Salience-SGG: Enhancing Unbiased Scene Graph Generation with Iterative Salience Estimation
Technische Universität Berlin（柏林技术大学）；Humboldt Universität zu Berlin（柏林洪堡大学）；Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK（格勒诺布尔阿尔卑斯大学、法国国家信息与自动化研究所、法国国家科学研究中心、格勒诺布尔INP、LJK）；Bernstein Center for Computational Neuroscience（计算神经科学伯恩斯坦中心）；Science of Intelligence Research Cluster of Excellence（卓越人工智能研究集群）
Monash University（莫纳什大学）；University of Electronic Science and Technology of China（电子科技大学）
Scene graph generation (SGG) is a fundamental task aimed at detecting visual relations between objects in an image. The prevailing SGG methods require all object classes to be given in the training set. Such a closed setting limits the practical application of SGG. In this paper, we introduce open-vocabulary scene graph generation, a novel, realistic and challenging setting in which a model is trained on a set of base object classes but is required to infer relations for unseen target object classes. To this end, we propose a two-step method that firstly pre-trains on large amounts of coarse-g…
University of Electronic Science and Technology of China（电子科技大学）；Sichuan Artificial Intelligence Research Institute（四川人工智能研究院）
Scene Graph Generation (SGG) represents objects and their interactions with a graph structure. Recently, many works are devoted to solving the imbalanced problem in SGG. However, underestimating the head predicates in the whole training process, they wreck the features of head predicates that provide general features for tail ones. Besides, assigning excessive attention to the tail predicates leads to semantic deviation. Based on this, we propose a novel SGG framework, learning to generate scene graphs from Head to Tail (SGG-HT), containing Curriculum Re-weight Mechanism (CRM) and Semantic Con…
VOST-SGG: VLM-Aided One-Stage Spatio-Temporal Scene Graph Generation
College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算与数据科学学院）；Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore（科学、技术与研究局高性能计算研究所）；Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore（科学、技术与研究局前沿人工智能研究中心）
Zhejiang University（浙江大学）；Columbia University（哥伦比亚大学）；University of Rochester（罗切斯特大学）
Current Scene Graph Generation (SGG) methods tend to predict frequent predicate categories and fail to recognize rare ones due to the severe imbalanced distribution of predicates. To improve the robustness of SGG models on different predicate categories, recent research has focused on unbiased SGG and adopted mean Recall@K (mR@K) as the main evaluation metric. However, we discovered two overlooked issues about this de facto standard metric, which makes current unbiased SGG evaluation vulnerable and unfair: 1) mR@K neglects the correlations among predicates and unintentionally breaks category i…
LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations
The Chinese University of Hong Kong（香港中文大学）；City University of Hong Kong（香港城市大学）；The University of Hong Kong（香港大学）；Dongguan University of Technology（东莞理工学院）
Scene graph generation (SGG) aims to capture a wide variety of interactions between pairs of objects, which is essential for full scene understanding. Existing SGG methods trained on the entire set of relations fail to acquire complex reasoning about visual and textual correlations due to various biases in training data. Learning on trivial relations that indicate generic spatial configuration like 'on' instead of informative relations such as 'parked on' does not enforce this complex reasoning, harming generalization. To address this problem, we propose a novel framework for SGG training that…
University of Electronic Science and Technology of China（电子科技大学）；Southwest University（西南大学）；Kuaishou（快手）
The scene graph generation (SGG) task aims to detect visual relationship triplets, i.e., subject, predicate, object, in an image, providing a structural vision layout for scene understanding. However, current models are stuck in common predicates, e.g., "on" and "at", rather than informative ones, e.g., "standing on" and "looking at", resulting in the loss of precise information and overall performance. If a model only uses "stone on road" rather than "blocking" to describe an image, it is easy to misunderstand the scene. We argue that this phenomenon is caused by two key imbalances between in…
University of Electronic Science and Technology of China（电子科技大学）；University of Wollongong（伍伦贡大学）；University of Trento（特伦托大学）；Northwestern Polytechnical University（西北工业大学）
Scene graph generation (SGG) is built on top of detected objects to predict object pairwise visual relations for describing the image content abstraction. Existing works have revealed that if the links between objects are given as prior knowledge, the performance of SGG is significantly improved. Inspired by this observation, in this article, we propose a relation regularized network (R2-Net), which can predict whether there is a relationship between two objects and encode this relation into object feature refinement and better SGG. Specifically, we first construct an affinity matrix among det…
Zhejiang University（浙江大学）；Columbia University（哥伦比亚大学）；University of Rochester（罗切斯特大学）
Unbiased SGG has achieved significant progress over recent years. However, almost all existing SGG models have overlooked the ground-truth annotation qualities of prevailing SGG datasets, i.e., they always assume: 1) all the manually annotated positive samples are equally correct; 2) all the un-annotated negative samples are absolutely background. In this paper, we argue that both assumptions are inapplicable to SGG: there are numerous "noisy" groundtruth predicate labels that break these two assumptions, and these noisy samples actually harm the training of unbiased SGG models. To this end, w…
Xidian University（西安电子科技大学）；National University of Defense Technology（国防科技大学）；Edith Cowan University（埃迪斯科文大学）；The University of Western Australia（西澳大学）
Deep learning techniques have led to remarkable breakthroughs in the field of generic object detection and have spawned a lot of scene-understanding tasks in recent years. Scene graph has been the focus of research because of its powerful semantic representation and applications to scene understanding. Scene Graph Generation (SGG) refers to the task of automatically mapping an image into a semantic structural scene graph, which requires the correct labeling of detected objects and their relationships. Although this is a challenging task, the community has proposed a lot of SGG approaches and a…
S^2Former-OR: Single-Stage Bi-Modal Transformer for Scene Graph Generation in OR
The Chinese University of Hong Kong（香港中文大学）；Institute of Medical Intelligence and XR（医学智能与XR研究院）；Technical University of Denmark（丹麦技术大学）；National University of Singapore（新加坡国立大学）
Scene graphs provide a rich, structured representation of a scene by encoding the entities (objects) and their spatial relationships in a graphical format. This representation has proven useful in several tasks, such as question answering, captioning, and even object detection, to name a few. Current approaches take a generation-by-classification approach where the scene graph is generated through labeling of all possible edges between objects in a scene, which adds computational overhead to the approach. This work introduces a generative transformer-based approach to generating scene graphs b…
Scene Graph Generation (SGG) aims to build a structured representation of a scene using objects and pairwise relationships, which benefits downstream tasks. However, current SGG methods usually suffer from sub-optimal scene graph generation because of the long-tailed distribution of training data. To address this problem, we propose Resistance Training using Prior Bias (RTPB) for the scene graph generation. Specifically, RTPB uses a distributed-based prior bias to improve models' detecting ability on less frequent relationships during training, thus improving the model generalizability on tail…
Dalian University of Technology（大连理工大学）；Meta Reality Labs（Meta现实实验室）；Dalian University（大连大学）；Beijing University of Technology（北京工业大学）
3D scene graph generation (SGG) has been of high interest in computer vision. Although the accuracy of 3D SGG on coarse classification and single relation label has been gradually improved, the performance of existing works is still far from being perfect for fine-grained and multi-label situations. In this paper, we propose a framework fully exploring contextual information for the 3D SGG task, which attempts to satisfy the requirements of fine-grained entity class, multiple relation labels, and high accuracy simultaneously. Our proposed approach is composed of a Graph Feature Extraction modu…
Scene graph generation (SGG) has gained tremendous progress in recent years. However, its underlying long-tailed distribution of predicate classes is a challenging problem. For extremely unbalanced predicate distributions, existing approaches usually construct complicated context encoders to extract the intrinsic relevance of scene context to predicates and complex networks to improve the learning ability of network models for highly imbalanced predicate distributions. To address the unbiased SGG problem, we introduce a simple yet effective method dubbed Context-Aware Mixture-of-Experts (CAME)…
From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）
JD Explore Academy（京东探索研究院）；Hangzhou Dianzi University（杭州电子科技大学）；The University of Sydney（悉尼大学）；Wuhan University（武汉大学）
Generating informative scene graphs from images requires integrating and reasoning from various graph components, i.e., objects and relationships. However, current scene graph generation (SGG) methods, including the unbiased SGG methods, still struggle to predict informative relationships due to the lack of 1) high-level inference such as transitive inference between relationships and 2) efficient mechanisms that can incorporate all interactions of graph components. To address the issues mentioned above, we devise a hyper-relationship learning network, termed HLN, for SGG. Specifically, the pr…
Monash University（莫纳什大学）；University of Electronic Science and Technology of China（电子科技大学）
Scene graphs provide valuable information to many downstream tasks. Many scene graph generation (SGG) models solely use the limited annotated relation triples for training, leading to their underperformance on low-shot (few and zero) scenarios, especially on the rare predicates. To address this problem, we propose a novel semantic compositional learning strategy that makes it possible to construct additional, realistic relation triples with objects from different images. Specifically, our strategy decomposes a relation triple by identifying and removing the unessential component and composes a…
Universidad Pablo de Olavide（巴勃罗·德·奥拉维德大学）；Universidad de Sevilla（塞维利亚大学）；Andalusian Research Institute in Data Science and Computational Intelligence (DaSCI)（安达卢西亚数据科学与计算智能研究所（DaSCI））；University of Granada（格拉纳达大学）
Scene graph generation from images is a task of great interest to applications such as robotics, because graphs are the main way to represent knowledge about the world and regulate human-robot interactions in tasks such as Visual Question Answering (VQA). Unfortunately, its corresponding area of machine learning is still relatively in its infancy, and the solutions currently offered do not specialize well in concrete usage scenarios. Specifically, they do not take existing "expert" knowledge about the domain world into account; and that might indeed be necessary in order to provide the level o…
Innovation Center Virtual Reality（虚拟现实创新中心）；ETH Zurich（苏黎世联邦理工学院）
Scene graphs are nodes and edges consisting of objects and object-object relationships, respectively. Scene graph generation (SGG) aims to identify the objects and their relationships. We propose a bidirectional GRU (BiGRU) transformer network (BGT-Net) for the scene graph generation for images. This model implements novel object-object communication to enhance the object information using a BiGRU layer. Thus, the information of all objects in the image is available for the other objects, which can be leveraged later in the object prediction step. This object information is used in a transform…
