In this paper, we address the problem of referring expression comprehension in videos, which is challenging due to complex expression and scene dynamics. Unlike previous methods which solve the problem in multiple stages (i.e., tracking, proposal-based matching), we tackle the problem from a novel perspective, \textbf{co-grounding}, with an elegant one-stage framework. We enhance the single-frame grounding accuracy by semantic attention learning and improve the cross-frame grounding consistency with co-grounding feature learning. Semantic attention learning explicitly parses referring cues in different attributes to reduce the ambiguity in the complex expression. Co-grounding feature learning boosts visual feature representations by integrating temporal correlation to reduce the ambiguity caused by scene dynamics. Experiment results demonstrate the superiority of our framework on the video grounding datasets VID and LiOTB in generating accurate and stable results across frames. Our model is also applicable to referring expression comprehension in images, illustrated by the improved performance on the RefCOCO dataset. Our project is available at https://sijiesong.github.io/co-grounding.
The University of Hong Kong（香港大学）；University of Wollongong（伍伦贡大学）；Tencent AI Lab（腾讯人工智能实验室）；Australian Centre for Robotic Vision, University of Adelaide（阿德莱德大学澳大利亚机器人视觉中心）
Northwestern Polytechnical University（西北工业大学）；National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（空天地海一体化大数据应用技术国家工程实验室）
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS（中国科学院计算技术研究所智能信息处理重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；University of Science and Technology of China（中国科学技术大学）；Peng Cheng Laboratory（鹏城实验室）
Referring Expression Comprehension Using Language Adaptive Inference
College of Computer Science & Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；Shanghai Institute for Advanced Study, Zhejiang University（浙江大学上海高等研究院）；Shanghai AI Laboratory（上海人工智能实验室）
Hierarchical Alignment-enhanced Adaptive Grounding Network for Generalized Referring Expression Comprehension
School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；Institute of Big Data, Fudan University（复旦大学大数据学院）；Shanghai University of Finance and Economics（上海财经大学）；Nanyang Technological University（南洋理工大学）
University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；University of Science and Technology of China（中国科学技术大学）；Nanjing University of Science and Technology（南京理工大学）；Huawei Cloud & AI（华为云与人工智能部门）
The University of Western Australia（西澳大利亚大学）；Murdoch University（莫道克大学）；Sun Yat-sen University（中山大学）；The Hong Kong University of Science and Technology（香港科技大学）
