We consider generation and comprehension of natural language referring expression for objects in an image. Unlike generic "image captioning" which lacks natural standard evaluation criteria, quality of a referring expression may be measured by the receiver's ability to correctly infer which object is being described. Following this intuition, we propose two approaches to utilize models trained for comprehension task to generate better expressions. First, we use a comprehension module trained on human-generated expressions, as a "critic" of referring expression generator. The comprehension module serves as a differentiable proxy of human evaluation, providing training signal to the generation module. Second, we use the comprehension module in a generate-and-rerank pipeline, which chooses from candidate expressions generated by a model according to their performance on the comprehension task. We show that both approaches lead to improved referring expression generation on multiple benchmark datasets.
The University of Hong Kong（香港大学）；University of Wollongong（伍伦贡大学）；Tencent AI Lab（腾讯人工智能实验室）；Australian Centre for Robotic Vision, University of Adelaide（阿德莱德大学澳大利亚机器人视觉中心）
Northwestern Polytechnical University（西北工业大学）；National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（空天地海一体化大数据应用技术国家工程实验室）
Referring Expression Comprehension Using Language Adaptive Inference
College of Computer Science & Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；Shanghai Institute for Advanced Study, Zhejiang University（浙江大学上海高等研究院）；Shanghai AI Laboratory（上海人工智能实验室）
Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension
School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；Institute of Big Data, Fudan University（复旦大学大数据学院）；Shanghai University of Finance and Economics（上海财经大学）；Nanyang Technological University（南洋理工大学）
The University of Western Australia（西澳大利亚大学）；Murdoch University（莫道克大学）；Sun Yat-sen University（中山大学）；The Hong Kong University of Science and Technology（香港科技大学）
ScanFormer: Referring Expression Comprehension by Iteratively Scanning
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；Zhejiang-Singapore Innovation and Al Joint Research Lab（浙新创新与人工智能联合研究实验室）
Continual Referring Expression Comprehension via Dual Modular Memorization
University of Electronic Science and Technology of China（电子科技大学）；Peng Cheng Laboratory（鹏城实验室）；University of Wollongong（伍伦贡大学）；Hefei University of Technology（合肥工业大学）
Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension
School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Zhejiang University-University of Illinois at Urbana-Champaign Institute, Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校研究院）
