首页/论文/2012.11691

arXiv 2012.116912020-12-21cs.CVcs.LG

Alleviating Noisy Data in Image Captioning with Cooperative Distillation

Pierre Dognin, Igor Melnyk, Youssef Mroueh, Inkit Padhi, Mattia Rigotti, Jarret Ross, Yair Schiff

备注：: CVPR 2020 VizWiz Challenge

英文摘要：

Image captioning systems have made substantial progress, largely due to the availability of curated datasets like Microsoft COCO or Vizwiz that have accurate descriptions of their corresponding images. Unfortunately, scarce availability of such cleanly labeled data results in trained algorithms producing captions that can be terse and idiosyncratically specific to details in the image. We propose a new technique, cooperative distillation that combines clean curated datasets with the web-scale automatically extracted captions of the Google Conceptual Captions dataset (GCC), which can have poor descriptions of images, but is abundant in size and therefore provides a rich vocabulary resulting in more expressive captions.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

超越通用：使用视觉-语言预训练模型融入真实世界知识以增强图像描述

Beyond Generic: Enhancing Image Captioning with Real-World Knowledge using Vision-Language Pre-Training Model

Nanjing University（南京大学）；University of Glasgow（格拉斯哥大学）

针对图像描述缺乏真实世界知识的问题，提出 Knowledge-guided Replay 方法，通过知识预测和蒸馏约束在微调中保留预训练知识，有效缓解幻觉，在 KnowCap 基准上显著提升知识识别准确率与描述质量。

2023-08-02cs.CLcs.CVcs.MM

Cooperative image captioning

Bar-Ilan University（巴伊兰大学）；NVIDIA research（英伟达研究院）

When describing images with natural language, the descriptions can be made more informative if tuned using downstream tasks. This is often achieved by training two networks: a "speaker network" that generates sentences given an image, and a "listener network" that uses them to perform a task. Unfortunately, training multiple networks jointly to communicate to achieve a joint task, faces two major challenges. First, the descriptions generated by a speaker network are discrete and stochastic, making optimization very hard and inefficient. Second, joint training usually causes the vocabulary used…

2019-07-26cs.CV

ControlCap: 可控的区域级图像描述生成

ControlCap: Controllable Region-level Captioning

University of Chinese Academy of Sciences（中国科学院大学）；Zhejiang University（浙江大学）；University of Virginia（弗吉尼亚大学）

针对区域级图像描述的退化问题，提出引入控制词的ControlCap方法，通过划分描述子空间约束生成范围以提升低频描述生成概率，支持交互式控制词增强泛化，在两个数据集上CIDEr得分大幅优于现有最优水平。

2024-03-09cs.CV

Non-Autoregressive Image Captioning with Counterfactuals-Critical Multi-Agent Learning

Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）

Most image captioning models are autoregressive, i.e. they generate each word by conditioning on previously generated words, which leads to heavy latency during inference. Recently, non-autoregressive decoding has been proposed in machine translation to speed up the inference time by generating all words in parallel. Typically, these models use the word-level cross-entropy loss to optimize each word independently. However, such a learning process fails to consider the sentence-level consistency, thus resulting in inferior generation quality of these non-autoregressive models. In this paper, we…

2020-05-10cs.CLcs.CVcs.LG

用于在无额外数据下提升图像描述能力的通信博弈涌现自然语言

Emergent Natural Language with Communication Games for Improving Image Captioning Capabilities without Additional Data

Department of Computer Science and Automation（计算机科学与自动化系）；Indian Institute of Science（印度科学院）

针对无监督图像描述性能不足的问题，提出多智能体强化学习博弈LoGIC，通过GRPO算法训练说话者与倾听者协作交流，在无额外标签下显著提升了图像描述性能。

2025-07-11cs.CVcs.LG

MotionAtlas: 面向运动中心视频的详细区域描述

MotionAtlas: Detailed Region Captioning for Motion-Centric Videos

Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Shanghai Jiao Tong University（上海交通大学）；Nanyang Technological University（南洋理工大学）；Peking University（北京大学）；Wuhan University（武汉大学）；Beijing Jiaotong University（北京交通大学）

提出MotionAtlas系统，通过区域感知运动描述、自举精炼数据管道和定制训练策略，在运动中心视频描述中显著提升细粒度理解，超越现有模型。

2026-06-28cs.AIcs.CV

更多相关论文收起更多论文

CoopInit: Initializing Generative Adversarial Networks via Cooperative Learning

Numerous research efforts have been made to stabilize the training of the Generative Adversarial Networks (GANs), such as through regularization and architecture design. However, we identify the instability can also arise from the fragile balance at the early stage of adversarial learning. This paper proposes the CoopInit, a simple yet effective cooperative learning-based initialization strategy that can quickly learn a good starting point for GANs, with a very small computation overhead during training. The proposed algorithm consists of two learning stages: (i) Cooperative initialization sta…

2023-03-21cs.CVeess.IV

CNN Fixations: An unraveling approach to visualize the discriminative image regions

Indian Institute of Science（印度科学学院）

Deep convolutional neural networks (CNN) have revolutionized various fields of vision research and have seen unprecedented adoption for multiple tasks such as classification, detection, captioning, etc. However, they offer little transparency into their inner workings and are often treated as black boxes that deliver excellent performance. In this work, we aim at alleviating this opaqueness of CNNs by providing visual explanations for the network's predictions. Our approach can analyze variety of CNN based models trained for vision applications such as object recognition and caption generation…

2018-12-12cs.CV

VolDoGer：用于视觉-语言任务中领域泛化的 LLM 辅助数据集

VolDoGer: LLM-assisted Datasets for Domain Generalization in Vision-Language Tasks

AITRICS Seoul（AITRICS首尔）；Chung-Ang University（Chung-ang 大学）

针对视觉-语言任务缺乏领域泛化数据集的问题，本文提出 VolDoGer，利用基于 LLM 的标注扩展方法构建覆盖图像描述、VQA 和视觉蕴含的基准，并评估从微调模型到多模态 LLM 的泛化能力。

2025-07-24cs.AIcs.CLcs.CV