Over the years, state-of-the-art (SoTA) image captioning methods have achieved promising results on some evaluation metrics (e.g., CIDEr). However, recent findings show that the captions generated by these methods tend to be biased toward the "average" caption that only captures the most general mode (a.k.a, language pattern) in the training corpus, i.e., the so-called mode collapse problem. Affected by it, the generated captions are limited in diversity and usually less informative than natural image descriptions made by humans. In this paper, we seek to avoid this problem by proposing a Discrete Mode Learning (DML) paradigm for image captioning. Our innovative idea is to explore the rich modes in the training caption corpus to learn a set of "mode embeddings", and further use them to control the mode of the generated captions for existing image captioning models. Specifically, the proposed DML optimizes a dual architecture that consists of an image-conditioned discrete variational autoencoder (CdVAE) branch and a mode-conditioned image captioning (MIC) branch. The CdVAE branch maps each image caption to one of the mode embeddings stored in a learned codebook, and is trained with a pure non-autoregressive generation objective to make the modes distinct and representative. The MIC branch can be simply modified from an existing image captioning model, where the mode embedding is added to the original word embeddings as the control signal. In the experiments, we apply the proposed DML to two widely used image captioning models, Transformer and AoANet. The results show that the learned mode embedding successfully facilitates these models to generate high-quality image captions with different modes, further leading to better performance for both diversity and quality on the MSCOCO dataset.
Diverse and Styled Image Captioning Using SVD-Based Mixture of Recurrent Experts
Amirkabir University of Technology (Tehran Polytechnic)（德黑兰理工大学）；Department of Mathematics and Computer Science（数学与计算机科学系）；Department of Computer Engineering（计算机工程系）
COMIC: Towards A Compact Image Captioning Model with Attention
University of Malaya（马来亚大学）
最近的工作在图像描述显示了非常有前途的原始性能。然而，我们意识到，大多数这些编码器-解码器风格的注意力网络不能自然地扩展到大的词汇量，使它们难以部署在硬件资源有限的嵌入式系统上。这是因为单词和输出嵌入矩阵的大小与词汇表的大小成比例地增长，从而对这些网络的紧凑性产生不利影响。针对这一局限性，本文在图像描述领域引入了一种全新的思想。也就是说，我们解决了迄今为止尚未探索的图像描述模型的紧凑性问题。我们发现，我们提出的模型，名为COMIC for COMpact图像描述，在MS-COCO和InstaPIC-1.1M数据集上，尽管嵌入词汇量小了39 - 99倍，但在五个常见的评估指标中，与最先进的方法取得了相当的结果。源代码和模型可在以下网址获得：https://github.com/jiahuei/COMIC-Compact-Image-Captioning-with-Attention
Neural Fashion Image Captioning : Accounting for Data Diversity
Ai4Innov；Masakhane NLP
图像描述的应用领域越来越广，时尚也不例外。拥有自动商品描述对于时尚网络平台来说非常有趣，有时候会托管数十万张图片。本文是第一个解决时尚图像的图像描述。为了解决数据集多样性问题，我们引入了InFashAIv1数据集，其中包含近16，000个非洲时尚商品图像及其标题，价格和一般描述。除了InFashAIv1之外，我们还使用了著名的DeepFashion数据集。字幕是使用由CNN编码器和RNN解码器组成的Show and Tell模型生成的。我们发现，在两个数据集上联合训练模型可以提高非洲风格时尚图像的字幕质量，这表明从西方风格数据中进行了迁移学习。InFashAIv1数据集在Github上发布，以鼓励更具多样性的作品。
Improving Image Captioning by Mimicking Human Reformulation Feedback at Inference-time
School of Computer Science and Engineering, The Hebrew University of Jerusalem（耶路撒冷希伯来大学计算机科学与工程学院）；School of Computing and Information Systems, University of Melbourne（墨尔本大学计算与信息系统学院）
FingerCap: Fine-grained Finger-level Hand Motion Captioning
Baidu Inc.（百度公司）；The University of Queensland（昆士兰大学）；Nanjing University（南京大学）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）；Peking University（北京大学）；Australian National University（澳大利亚国立大学）；City University of Macau（澳门城市大学）
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；The University of Hong Kong（香港大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
FIOVA: A Multi-Annotator Benchmark for Human-Aligned Video Captioning
Nanyang Technological University（南洋理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Zhongguancun Academy（中关村学院）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Institute of Technology（北京理工大学）；Southeast University（东南大学）；University of Science and Technology Beijing（北京科技大学）
TRECVID 2019: An Evaluation Campaign to Benchmark Video Activity Detection, Video Captioning and Matching, and Video Search & Retrieval
Georgetown University（乔治敦大学）；NIST（美国国家标准与技术研究院）；Johns Hopkins University（约翰斯·霍普金斯大学）；Guest Researcher, NIST（美国国家标准与技术研究院客座研究员）；Dakota-consulting（达科他咨询公司）；Insight Research Centre, Dublin City University（都柏林城市大学洞察研究中心）；ADAPT Research Centre, Dublin City University（都柏林城市大学ADAPT研究中心）；Leiden University（莱顿大学）；TNO（荷兰应用科学研究组织）；Laboratoire d’Informatique de Grenoble（格勒诺布尔计算机实验室）；National Institute of Standards and Technology（美国国家标准与技术研究院）；Information Access Division（信息获取部）
FusionRS: A Large-Scale RGB-Infrared-Style Remote Sensing Dataset for Cross-Modal Vision-Language Learning
China University of Petroleum-Beijing at Karamay（中国石油大学（北京）克拉玛依校区）；University of Electronic Science and Technology of China（电子科技大学）；Tianjin University（天津大学）
Automatic Identification and Description of Jewelry Through Computer Vision and Neural Networks for Translators and Interpreters
Department of Computer Science, University of Córdoba（计算机科学系，科尔多瓦大学）；Economics, Engineering, Society and Business Organization, University of Tuscia（经济学、工程学、社会与商业组织，图斯卡大学）；Department of Social Sciences, Translation and Interpreting, University of Córdoba（社会科学系，翻译与口译，科尔多瓦大学）
