Compare and Reweight: Distinctive Image Captioning Using Similar Images Sets
City University of Hong Kong（香港城市大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Jiuniu Wang, Wenjia Xu, Qingzhong Wang, Antoni B. Chan
A wide range of image captioning models has been developed, achieving significant improvement based on popular metrics, such as BLEU, CIDEr, and SPICE. However, although the generated captions can accurately describe the image, they are generic for similar images and lack distinctiveness, i.e., cannot properly describe the uniqueness of each image. In this paper, we aim to improve the distinctiveness of image captions through training with sets of similar images. First, we propose a distinctiveness metric -- between-set CIDEr (CIDErBtw) to evaluate the distinctiveness of a caption with respect to those of similar images. Our metric shows that the human annotations of each image are not equivalent based on distinctiveness. Thus we propose several new training strategies to encourage the distinctiveness of the generated caption for each image, which are based on using CIDErBtw in a weighted loss function or as a reinforcement learning reward. Finally, extensive experiments are conducted, showing that our proposed approach significantly improves both distinctiveness (as measured by CIDErBtw and retrieval metrics) and accuracy (e.g., as measured by CIDEr) for a wide variety of image captioning baselines. These results are further confirmed through a user study.
Group-based Distinctive Image Captioning with Memory Attention
City University of Hong Kong（香港城市大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Baidu Research（百度研究院）
Distinctive Image Captioning via CLIP Guided Group Optimization
McGill University（麦吉尔大学）；City University of Hong Kong（香港城市大学）；South China University of Technology（华南理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
On Distinctive Image Captioning via Comparing and Reweighting
City University of Hong Kong（香港城市大学）；Baidu Research, Baidu Inc.（百度公司百度研究院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Diverse and Styled Image Captioning Using SVD-Based Mixture of Recurrent Experts
Amirkabir University of Technology (Tehran Polytechnic)（德黑兰理工大学）；Department of Mathematics and Computer Science（数学与计算机科学系）；Department of Computer Engineering（计算机工程系）
Sam-Guided Enhanced Fine-Grained Encoding with Mixed Semantic Learning for Medical Image Captioning
Zhejiang University-University of Illinois Urbana-Champaign Institute, Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合学院，浙江大学）；Sir Run Run Shaw Hospital, Zhejiang University School of Medicine（浙江大学医学院附属邵逸夫医院）
University of Science and Technology of China（中国科学技术大学）；College of Software, Beihang University（北京航空航天大学软件学院）；SenseTime Research（商汤科技研究院）；The Chinese Academy of Sciences（中国科学院）；CUHK-SenseTime Joint Lab（香港中文大学-商汤科技联合实验室）；The Chinese University of Hong Kong（香港中文大学）；Key Laboratory of Electromagnetic Space Information（电磁空间信息重点实验室）
