Chengxi Li, Brent Harrison
In this paper, we build two automatic evaluation metrics for evaluating the association between a machine-generated caption and a ground truth stylized caption: OnlyStyle and StyleCIDEr.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
Stylized image captioning systems aim to generate a caption not only semantically related to a given image but also consistent with a given style description. One of the biggest challenges with this task is the lack of sufficient paired stylized data. Many studies focus on unsupervised approaches, without considering from the perspective of data augmentation. We begin with the observation that people may recall similar emotions when they are in similar scenes, and often express similar emotions with similar style phrases, which underpins our data augmentation idea. In this paper, we propose a …
Generating stylized captions for an image is an emerging topic in image captioning. Given an image as input, it requires the system to generate a caption that has a specific style (e.g., humorous, romantic, positive, and negative) while describing the image content semantically accurately. In this paper, we propose a novel stylized image captioning model that effectively takes both requirements into consideration. To this end, we first devise a new variant of LSTM, named style-factual LSTM, as the building block of our model. It uses two groups of matrices to capture the factual and stylized k…
Stylized image captioning as presented in prior work aims to generate captions that reflect characteristics beyond a factual description of the scene composition, such as sentiments. Such prior work relies on given sentiment identifiers, which are used to express a certain global style in the caption, e.g. positive or negative, however without taking into account the stylistic content of the visual scene. To address this shortcoming, we first analyze the limitations of current stylized captioning datasets and propose COCO attribute-based augmentations to obtain varied stylized captions from CO…
Large VLM-based Stylized Sports Captioning
本文针对现有大型视觉语言模型在体育图像描述中缺乏领域术语和风格化能力的问题，提出一种两级微调LVLM流水线，在F1和BERT分数上显著提升，并在超级碗直播中验证了其高效实用性。
With great advances in vision and natural language processing, the generation of image captions becomes a need. In a recent paper, Mathews, Xie and He [1], extended a new model to generate styled captions by separating semantics and style. In continuation of this work, here a new captioning model is developed including an image encoder to extract the features, a mixture of recurrent networks to embed the set of extracted features to a set of words, and a sentence generator that combines the obtained words as a stylized sentence. The resulted system that entitled as Mixture of Recurrent Experts…
Image captioning as a multimodal task has drawn much interest in recent years. However, evaluation for this task remains a challenging problem. Existing evaluation metrics focus on surface similarity between a candidate caption and a set of reference captions, and do not check the actual relation between a caption and the underlying visual content. We introduce a new diagnostic evaluation framework for the task of image captioning, with the goal of directly assessing models for grammaticality, truthfulness and diversity (GTD) of generated captions. We demonstrate the potential of our evaluatio…
Despite the remarkable progress of image captioning, existing captioners typically lack the controllable capability to generate desired image captions, e.g., describing the image in a rough or detailed manner, in a factual or emotional view, etc. In this paper, we show that a unified model is qualified to perform well in diverse domains and freely switch among multiple styles. Such a controllable capability is achieved by embedding the prompt learning into the image captioning framework. To be specific, we design a set of prompts to fine-tune the pre-trained image captioner. These prompts allo…
While most image captioning aims to generate objective descriptions of images, the last few years have seen work on generating visually grounded image captions which have a specific style (e.g., incorporating positive or negative sentiment). However, because the stylistic component is typically the last part of training, current models usually pay more attention to the style at the expense of accurate content description. In addition, there is a lack of variability in terms of the stylistic aspects. To address these issues, we propose an image captioning model called ATTEND-GAN which has two c…
This paper presents a Keyword-driven and N-gram Graph based approach for Image Captioning (KENGIC). Most current state-of-the-art image caption generators are trained end-to-end on large scale paired image-caption datasets which are very laborious and expensive to collect. Such models are limited in terms of their explainability and their applicability across different domains. To address these limitations, a simple model based on N-Gram graphs which does not require any end-to-end training on paired image captions is proposed. Starting with a set of image keywords considered as nodes, the gen…
SEM-POS: Grammatically and Semantically Correct Video Captioning
本文提出融合词性成分与视觉-空间特征的全局-局部融合网络SEM-POS，通过GLFB和POS监督提升视频字幕的语法与语义正确性，并在MSVD、MSRVTT上达到新SOTA。
Modern image captioning models are usually trained with text similarity objectives. However, since reference captions in public datasets often describe the most salient common objects, models trained with text similarity objectives tend to ignore specific and detailed aspects of an image that distinguish it from others. Toward more descriptive and distinctive caption generation, we propose using CLIP, a multimodal encoder trained on huge image-text pairs from web, to calculate multimodal similarity and use it as a reward function. We also propose a simple finetuning strategy of the CLIP text e…
Encoder-Decoder Based Long Short-Term Memory (LSTM) Model for Video Captioning
本研究提出一种基于编码器-解码器LSTM的视频字幕生成模型，通过将视频帧时间序列映射为单词序列生成字幕，并使用2-gram BLEU分数评估其正确性与泛化能力。
RONA: Pragmatically Diverse Image Captioning with Coherence Relations
本文提出 RONA 提示策略，将连贯关系作为 MLLM 生成图像描述时的语用变化控制轴，从而在多个领域提升描述多样性及与真值的对齐程度。
JaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models
针对现有n元语法类图像字幕评估指标与人类评估相关性低、日语缺乏等效指标的问题，提出基于场景图的日语评估指标JaSPICE，实验证明其与人类评估的相关性优于基线指标。
Image Captioning is a current research task to describe the image content using the objects and their relationships in the scene. To tackle this task, two important research areas converge, artificial vision, and natural language processing. In Image Captioning, as in any computational intelligence task, the performance metrics are crucial for knowing how well (or bad) a method performs. In recent years, it has been observed that classical metrics based on n-grams are insufficient to capture the semantics and the critical meaning to describe the content in an image. Looking to measure how well…
Progress in image captioning is gradually getting complex as researchers try to generalized the model and define the representation between visual features and natural language processing. This work tried to define such kind of relationship in the form of representation called Tensor Product Representation (TPR) which generalized the scheme of language modeling and structuring the linguistic attributes (related to grammar and parts of speech of language) which will provide a much better structure and grammatically correct sentence. TPR enables better and unique representation and structuring o…
ADS-Cap: A Framework for Accurate and Diverse Stylized Captioning with Unpaired Stylistic Corpora
提出ADS-Cap框架，通过对比学习模块、条件变分自编码器和复核模块，利用非配对风格语料库生成准确且多样化的风格化图像描述，在多个数据集上表现优异。
Automatic video captioning aims for a holistic visual scene understanding. It requires a mechanism for capturing temporal context in video frames and the ability to comprehend the actions and associations of objects in a given timeframe. Such a system should additionally learn to abstract video sequences into sensible representations as well as to generate natural written language. While the majority of captioning models focus solely on the visual inputs, little attention has been paid to the audiovisual modality. To tackle this issue, we propose a novel two-fold approach. First, we implement …
In image captioning where fluency is an important factor in evaluation, e.g., $n$-gram metrics, sequential models are commonly used; however, sequential models generally result in overgeneralized expressions that lack the details that may be present in an input image. Inspired by the idea of the compositional neural module networks in the visual question answering task, we introduce a hierarchical framework for image captioning that explores both compositionality and sequentiality of natural language. Our algorithm learns to compose a detail-rich sentence by selectively attending to different …
Video captioning aims to describe events in a video with natural language. In recent years, many works have focused on improving captioning models' performance. However, like other text generation tasks, it risks introducing factual errors not supported by the input video. These factual errors can seriously affect the quality of the generated text, sometimes making it completely unusable. Although factual consistency has received much research attention in text-to-text tasks (e.g., summarization), it is less studied in the context of vision-based text generation. In this work, we conduct a det…
Surprisal reveals diversity gaps in image captioning and different scorers change the story
本文提出用意外度方差衡量图像描述多样性，发现不同评分器会反转人类与模型之间的多样性比较结论，强调多样性评估需采用多评分器报告。
This paper explores new evaluation perspectives for image captioning and introduces a noun translation task that achieves comparative image caption generation performance by translating from a set of nouns to captions. This implies that in image captioning, all word categories other than nouns can be evoked by a powerful language model without sacrificing performance on n-gram precision. The paper also investigates lower and upper bounds of how much individual word categories in the captions contribute to the final BLEU score. A large possible improvement exists for nouns, verbs, and prepositi…
Recently, Reinforcement Learning (RL) approaches have demonstrated advanced performance in image captioning by directly optimizing the metric used for testing. However, this shaped reward introduces learning biases, which reduces the readability of generated text. In addition, the large sample space makes training unstable and slow. To alleviate these issues, we propose a simple coherent solution that constrains the action space using an n-gram language prior. Quantitative and qualitative evaluations on benchmarks show that RL with the simple add-on module performs favorably against its counte…
Image captioning is an ambiguous problem, with many suitable captions for an image. To address ambiguity, beam search is the de facto method for sampling multiple captions. However, beam search is computationally expensive and known to produce generic captions. To address this concern, some variational auto-encoder (VAE) and generative adversarial net (GAN) based methods have been proposed. Though diverse, GAN and VAE are less accurate. In this paper, we first predict a meaningful summary of the image, then generate the caption based on that summary. We use part-of-speech as summaries, since o…
Visual Captioning at Will: Describing Images and Videos Guided by a Few Stylized Sentences
本文提出少样本风格化视觉描述框架FS-StyleCap，利用条件编码器-解码器语言模型和视觉投影模块，通过两步训练方案在无标签文本语料上学习风格表示，仅需少量样本即可在推理时生成任意风格的图像或视频描述，自动评估超越现有方法且媲美全监督模型。
In this work, we have introduced Gaussian Smoothen Semantic Features (GSSF) for Better Semantic Selection for Indian regional language-based image captioning and introduced a procedure where we used the existing translation and English crowd-sourced sentences for training. We have shown that this architecture is a promising alternative source, where there is a crunch in resources. Our main contribution of this work is the development of deep learning architectures for the Bengali language (is the fifth widely spoken language in the world) with a completely different grammar and language attrib…
CIAN: Multi-Stage Framework for Event-Enriched Image Captioning via Retrieval-Augmented Generation
提出多阶段框架CIAN，通过检索相关文章并利用LoRA微调Qwen模型生成叙事，结合N-Gram精炼，在OpenEvents-V1上提升CIDEr从0.030到0.094，实现事件丰富的图像描述。
Live Video Captioning
本文提出实时视频字幕生成（LVC）新范式，结合可变形变换器与时间滤波在线生成字幕，并设计新评估指标，在ActivityNet Captions上超越离线方法。
Image captioning models have been able to generate grammatically correct and human understandable sentences. However most of the captions convey limited information as the model used is trained on datasets that do not caption all possible objects existing in everyday life. Due to this lack of prior information most of the captions are biased to only a few objects present in the scene, hence limiting their usage in daily life. In this paper, we attempt to show the biased nature of the currently existing image captioning models and present a new image captioning dataset, Egoshots, consisting of …
With the maturity of visual detection techniques, we are more ambitious in describing visual content with open-vocabulary, fine-grained and free-form language, i.e., the task of image captioning. In particular, we are interested in generating longer, richer and more fine-grained sentences and paragraphs as image descriptions. Image captioning can be translated to the task of sequential language prediction given visual content, where the output sequence forms natural language description with plausible grammar. However, existing image captioning methods focus only on language policy while not v…
Fluent and Accurate Image Captioning with a Self-Trained Reward Model
本文提出Self-Cap方法，通过基于自生成负样本训练的可学习判别器作为奖励模型，在提升图像描述正确性与丰富度的同时避免CLIP奖励导致的语法问题，并缩短了微调时间。
The interest in Artificial Intelligence (AI) and its applications has seen unprecedented growth in the last few years. The success can be partly attributed to the advancements of deep neural networks made in the sub-fields of AI such as Computer Vision (CV) and Natural Language Processing (NLP). The promising research area that this dissertation focuses on is visual and language understanding which involves many challenging tasks, i.e., classification, detection, segmentation, machine translation and captioning, etc. The state-of-the-art methods for solving these problems usually involves only…
UnMA-CapSumT: Unified and Multi-Head Attention-driven Caption Summarization Transformer
本文提出UnMA-CapSumT框架，结合事实性与风格化（浪漫、幽默）图像描述，通过多头注意力摘要Transformer解决词汇外和重复问题，在Flickr8K和FlickrStyle10K上验证有效性。
The recent progress on image recognition and language modeling is making automatic description of image content a reality. However, stylized, non-factual aspects of the written description are missing from the current systems. One such style is descriptions with emotions, which is commonplace in everyday communication, and influences decision-making and interpersonal relationships. We design a system to describe an image with emotions, and present a model that automatically generates captions with positive or negative sentiments. We propose a novel switching recurrent neural network with word-…
CAPTCHAs based on reading text are susceptible to machine-learning-based attacks due to recent significant advances in deep learning (DL). To address this, this paper promotes image/visual captioning based CAPTCHAs, which is robust against machine-learning-based attacks. To develop image/visual-captioning-based CAPTCHAs, this paper proposes a new image captioning architecture by exploiting tensor product representations (TPR), a structured neural-symbolic framework developed in cognitive science over the past 20 years, with the aim of integrating DL with explicit language structures and rules.…
VisualQuest: A Benchmark for Abstract Visual Reasoning in MLLMs
VisualQuest通过风格化图像和针对性问题，评估多模态大语言模型在抽象视觉推理上的能力，发现Gemini和GPT-4o在不同任务上表现突出，揭示多模态理解的挑战。
Image Captioning is a task that requires models to acquire a multi-modal understanding of the world and to express this understanding in natural language text. While the state-of-the-art for this task has rapidly improved in terms of n-gram metrics, these models tend to output the same generic captions for similar images. In this work, we address this limitation and train a model that generates more diverse and specific captions through an unsupervised training approach that incorporates a learning signal from an Image Retrieval model. We summarize previous results and improve the state-of-the…
Image captioning has demonstrated models that are capable of generating plausible text given input images or videos. Further, recent work in image generation has shown significant improvements in image quality when text is used as a prior. Our work ties these concepts together by creating an architecture that can enable bidirectional generation of images and text. We call this network Multi-Modal Vector Representation (MMVR). Along with MMVR, we propose two improvements to the text conditioned image generation. Firstly, a n-gram metric based cost function is introduced that generalizes the cap…
Satellite Captioning: Large Language Models to Augment Labeling
针对遥感图像等特定领域字幕数据集存在的词汇有限、语法错误等问题，本研究以RSICD数据集为对象，验证了ChatGPT语法校正可有效提升字幕模型性能准确率。
DescribeEarth: Describe Anything for Remote Sensing Images
针对遥感图像缺乏对象级细粒度描述的问题，本文提出 Geo-DLC 任务、DE-Dataset、DE-Benchmark 及 DescribeEarth 模型，并验证其优于通用 MLLM。
In recent years, vision and language pre-training (VLP) models have advanced the state-of-the-art results in a variety of cross-modal downstream tasks. Aligning cross-modal semantics is claimed to be one of the essential capabilities of VLP models. However, it still remains unclear about the inner working mechanism of alignment in VLP models. In this paper, we propose a new probing method that is based on image captioning to first empirically study the cross-modal semantics alignment of VLP models. Our probing method is built upon the fact that given an image-caption pair, the VLP models will …
Towards L-System Captioning for Tree Reconstruction
本文提出一种从图像直接推断L系统单词表示的新方法，用于树木重建，端到端训练模型，在二维拓扑上验证，可避免点云提取并无需预定义语法。
Recent progress in style transfer on images has focused on improving the quality of stylized images and speed of methods. However, real-time methods are highly unstable resulting in visible flickering when applied to videos. In this work we characterize the instability of these methods by examining the solution set of the style transfer objective. We show that the trace of the Gram matrix representing style is inversely related to the stability of the method. Then, we present a recurrent convolutional network for real-time video style transfer which incorporates a temporal consistency loss and…
DynaStride: Dynamic Stride Windowing with MMCoT for Instructional Multi-Scene Captioning
DynaStride通过多模态链式思考和动态步长窗算法，实现无需手动分割的多场景教学视频连贯描述生成，提升描述的连贯性和信息量。
L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training
提出轻量级描述指标L-CLIPScore，通过权重复用和矩阵分解压缩双编码器，并设计多模态相似度调节器损失进行蒸馏，在保持多模态对齐能力的同时降低计算成本，可用于高效评估描述质量及指导模型训练。
The open-ended nature of visual captioning makes it a challenging area for evaluation. The majority of proposed models rely on specialized training to improve human-correlation, resulting in limited adoption, generalizability, and explainabilty. We introduce "typicality", a new formulation of evaluation rooted in information theory, which is uniquely suited for problems lacking a definite ground truth. Typicality serves as our framework to develop a novel semantic comparison, SPARCS, as well as referenceless fluency evaluation metrics. Over the course of our analysis, two separate dimensions o…
In this paper, we build a multi-style generative model for stylish image captioning which uses multi-modality image features, ResNeXt features and text features generated by DenseCap. We propose the 3M model, a Multi-UPDOWN caption model that encodes multi-modality features and decode them to captions. We demonstrate the effectiveness of our model on generating human-like captions by examining its performance on two datasets, the PERSONALITY-CAPTIONS dataset and the FlickrStyle10K dataset. We compare against a variety of state-of-the-art baselines on various automatic NLP metrics such as BLEU,…
Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Training
针对现有字幕评估指标的缺陷，提出基于CLIP的可学习指标PAC-S++，通过正样本增强对比学习优化，可用于评估及作为奖励微调字幕模型，提升生成字幕质量。
Automatic radiology report generation has been an attracting research problem towards computer-aided diagnosis to alleviate the workload of doctors in recent years. Deep learning techniques for natural image captioning are successfully adapted to generating radiology reports. However, radiology image reporting is different from the natural image captioning task in two aspects: 1) the accuracy of positive disease keyword mentions is critical in radiology image reporting in comparison to the equivalent importance of every single word in a natural image caption; 2) the evaluation of reporting qua…
SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluation
针对长图像描述评估指标成本高或相关性低的问题，提出无参考指标 SPECS，通过修改 CLIP 奖励正确细节并惩罚错误细节，在匹配开源 LLM 指标性能的同时大幅提升效率。
The MSR-Video to Text Dataset with Clean Annotations
本文针对MSR-VTT数据集中标注存在重复和语法噪声的问题进行数据清洗，实验证明清洗后的数据能提升典型视频字幕生成模型的定量指标和人类评价中的连贯性与相关性。
Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Models
提出Image2Text2Image框架，利用扩散模型将图像描述生成新图像并与原图比较相似度，实现无需人工标注的图像描述模型评估。
A Novel Evaluation Framework for Image2Text Generation
本文提出一种基于具备图像生成能力的 LLM 的无参考图像描述评估框架，通过“图像—描述—再生成图像”的相似度衡量模型性能，并经人工评估验证有效。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。