Change Captioning is a task that aims to describe the difference between images with natural language. Most existing methods treat this problem as a difference judgment without the existence of distractors, such as viewpoint changes. However, in practice, viewpoint changes happen often and can overwhelm the semantic difference to be described. In this paper, we propose a novel visual encoder to explicitly distinguish viewpoint changes from semantic changes in the change captioning task. Moreover, we further simulate the attention preference of humans and propose a novel reinforcement learning process to fine-tune the attention directly with language evaluation rewards. Extensive experimental results show that our method outperforms the state-of-the-art approaches by a large margin in both Spot-the-Diff and CLEVR-Change datasets.
EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching
University of Science and Technology of China(中国科学技术大学);Southeast University(东南大学);Alibaba Group(阿里巴巴集团);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心)
Large Language Models for Captioning and Retrieving Remote Sensing Images
INESC-ID, Instituto Superior Técnico, University of Lisbon(里斯本大学高等技术学院INESC-ID研究所);Universidade NOVA de Lisboa(新里斯本大学);Ecole Polytechnique Fédérale de Lausanne(洛桑联邦理工学院)
Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning
University of Chinese Academy of Sciences(中国科学院大学);Hangzhou Dianzi University (HDU)(杭州电子科技大学);Lishui Institute of HDU(杭州电子科技大学丽水研究院);Key Laboratory of AI Safety of CAS(中国科学院人工智能安全重点实验室);Institute of Computing Technology, Chinese Academy of Sciences (CAS)(中国科学院计算技术研究所)
Neighborhood Contrastive Transformer for Change Captioning
University of Chinese Academy of Sciences(中国科学院大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Peng Cheng Laboratory(鹏城实验室)
Self-supervised Cross-view Representation Reconstruction for Change Captioning
University of Chinese Academy of Sciences(中国科学院大学);Key Lab of Intelligent Information Processing, ICT, CAS(中国科学院计算技术研究所智能信息处理重点实验室);Lishui Institute of Hangzhou Dianzi University(杭州电子科技大学丽水研究院);Peng Cheng Laboratory(鹏城实验室);University of Science and Technology of China(中国科学技术大学);Hangzhou Dianzi University(杭州电子科技大学)
The Language of Touch: Translating Vibrations into Text with Dual-Branch Learning
Fujian Key Lab for Intelligent Processing and Wireless Transmission of Media Information, Fuzhou University(福州大学福建省媒体信息智能处理与无线传输重点实验室);Fujian Science and Technology Innovation Laboratory for Optoelectronic Information of China(福建省光电信息科学与技术创新实验室);School of Artificial Intelligence, Hubei University(湖北大学人工智能学院);Key Laboratory of Intelligent Sensing System and Security, Hubei University and Ministry of Education(湖北大学及教育部智能感知系统与安全重点实验室);School of Computer Science and Engineering, South China University of Technology(华南理工大学计算机科学与工程学院)