In visual relationship detection, human-notated relationships can be regarded as determinate relationships. However, there are still large amount of unlabeled data, such as object pairs with less significant relationships or even with no relationships. We refer to these unlabeled but potentially useful data as undetermined relationships. Although a vast body of literature exists, few methods exploit these undetermined relationships for visual relationship detection.
In this paper, we explore the beneficial effect of undetermined relationships on visual relationship detection. We propose a novel multi-modal feature based undetermined relationship learning network (MF-URLN) and achieve great improvements in relationship detection. In detail, our MF-URLN automatically generates undetermined relationships by comparing object pairs with human-notated data according to a designed criterion. Then, the MF-URLN extracts and fuses features of object pairs from three complementary modals: visual, spatial, and linguistic modals. Further, the MF-URLN proposes two correlated subnetworks: one subnetwork decides the determinate confidence, and the other predicts the relationships. We evaluate the MF-URLN on two datasets: the Visual Relationship Detection (VRD) and the Visual Genome (VG) datasets. The experimental results compared with state-of-the-art methods verify the significant improvements made by the undetermined relationships, e.g., the top-50 relation detection recall improves from 19.5% to 23.9% on the VRD dataset.
Exploring Learning Models for Topological Relationship Recognition from Image Data
Indian Institute of Science Education and Research, Thiruvanantapuram(印度科学教育与研究学院蒂鲁文南特普拉姆校区);Indian Institute of Science Education and Research, Kolkata(印度科学教育与研究学院加尔各答校区)
Exploring Long Tail Visual Relationship Recognition with Large Vocabulary
King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学);IIT Roorkee(印度理工学院鲁尔基分校);Stanford University(斯坦福大学);Baidu(百度);NTU Singapore(新加坡南洋理工大学)
Exploring Contextual Relationships for Cervical Abnormal Cell Detection
Central South University(中南大学);The Second Xiangya Hospital, Central South University(中南大学湘雅二医院);The Fourth Hospital of Hebei Medical University(河北医科大学第四医院)
Exploring Partial Multi-Label Learning via Integrating Semantic Co-occurrence Knowledge
School of Computing and Artificial Intelligence, Southwest Jiaotong University(计算机与人工智能学院,西南交通大学);Engineering Research Center of Sustainable Urban Intelligent Transportation, Ministry of Education(可持续城市智能交通工程研究中心,教育部);School of Engineering, Swinburne University of Technology(工程学院,斯winburne大学);School of Electronic and Information Engineering, Wuyi University(电子与信息工程学院,五邑大学);College of Electrical Engineering, Sichuan University(电气工程学院,四川大学);School of Computer Science, Chengdu University(计算机科学学院,成都大学)
Things not Written in Text: Exploring Spatial Commonsense from Visual Signals
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);The MOE Key Laboratory of Computational Linguistics, Peking University(北京大学计算语言学教育部重点实验室);Artificial Intelligence Institute of Peking University(北京大学人工智能研究院);State Key Laboratory of Media Convergence Production Technology and Systems(媒体融合生产技术与系统国家重点实验室)