Referring expression comprehension aims to localize the object instance described by a natural language expression. Current referring expression methods have achieved good performance. However, none of them is able to achieve real-time inference without accuracy drop. The reason for the relatively slow inference speed is that these methods artificially split the referring expression comprehension into two sequential stages including proposal generation and proposal ranking. It does not exactly conform to the habit of human cognition. To this end, we propose a novel Realtime Cross-modality Correlation Filtering method (RCCF). RCCF reformulates the referring expression comprehension as a correlation filtering process. The expression is first mapped from the language domain to the visual domain and then treated as a template (kernel) to perform correlation filtering on the image feature map. The peak value in the correlation heatmap indicates the center points of the target box. In addition, RCCF also regresses a 2-D object size and 2-D offset. The center point coordinates, object size and center point offset together to form the target bounding box. Our method runs at 40 FPS while achieving leading performance in RefClef, RefCOCO, RefCOCO+ and RefCOCOg benchmarks. In the challenging RefClef dataset, our methods almost double the state-of-the-art performance (34.70% increased to 63.79%). We hope this work can arouse more attention and studies to the new cross-modality correlation filtering framework as well as the one-stage framework for referring expression comprehension.
The University of Hong Kong（香港大学）；University of Wollongong（伍伦贡大学）；Tencent AI Lab（腾讯人工智能实验室）；Australian Centre for Robotic Vision, University of Adelaide（阿德莱德大学澳大利亚机器人视觉中心）
Northwestern Polytechnical University（西北工业大学）；National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（空天地海一体化大数据应用技术国家工程实验室）
Referring Expression Comprehension Using Language Adaptive Inference
College of Computer Science & Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；Shanghai Institute for Advanced Study, Zhejiang University（浙江大学上海高等研究院）；Shanghai AI Laboratory（上海人工智能实验室）
Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension
School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；Institute of Big Data, Fudan University（复旦大学大数据学院）；Shanghai University of Finance and Economics（上海财经大学）；Nanyang Technological University（南洋理工大学）
The University of Western Australia（西澳大利亚大学）；Murdoch University（莫道克大学）；Sun Yat-sen University（中山大学）；The Hong Kong University of Science and Technology（香港科技大学）
ScanFormer: Referring Expression Comprehension by Iteratively Scanning
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；Zhejiang-Singapore Innovation and Al Joint Research Lab（浙新创新与人工智能联合研究实验室）
Continual Referring Expression Comprehension via Dual Modular Memorization
University of Electronic Science and Technology of China（电子科技大学）；Peng Cheng Laboratory（鹏城实验室）；University of Wollongong（伍伦贡大学）；Hefei University of Technology（合肥工业大学）
