News captioning task aims to generate sentences by describing named entities or concrete events for an image with its news article. Existing methods have achieved remarkable results by relying on the large-scale pre-trained models, which primarily focus on the correlations between the input news content and the output predictions. However, the news captioning requires adhering to some fundamental rules of news reporting, such as accurately describing the individuals and actions associated with the event. In this paper, we propose the rule-driven news captioning method, which can generate image descriptions following designated rule signal. Specifically, we first design the news-aware semantic rule for the descriptions. This rule incorporates the primary action depicted in the image (e.g., "performing") and the roles played by named entities involved in the action (e.g., "Agent" and "Place"). Second, we inject this semantic rule into the large-scale pre-trained model, BART, with the prefix-tuning strategy, where multiple encoder layers are embedded with news-aware semantic rule. Finally, we can effectively guide BART to generate news sentences that comply with the designated rule. Extensive experiments on two widely used datasets (i.e., GoodNews and NYTimes800k) demonstrate the effectiveness of our method.
How to Understand Named Entities: Using Common Sense for News Captioning
School of Electrical and Information Engineering, Tianjin University(天津大学电气与信息工程学院);School of Computing, National University of Singapore(新加坡国立大学计算机学院)
EAMA : Entity-Aware Multimodal Alignment Based Approach for News Image Captioning
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);School of Electronics Engineering and Computer Science, Peking University(北京大学电子工程与计算机科学学院)
Enhancing Journalism with AI: A Study of Contextualized Image Captioning for News Articles using LLMs and LMMs
DFKI German Research Center for Artificial Intelligence(德国人工智能研究中心);Applied Artificial Intelligence, Carl von Ossietzky University Oldenburg(奥尔登堡卡尔·冯·奥西茨基大学应用人工智能系)
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学)
图像描述传统上依赖于基于参考的自动评估,其中将机器字幕与人类书写的字幕进行比较。这与人类评估字幕质量的无参考方式形成对比。在本文中,我们报告了令人惊讶的实证发现,CLIP(Radford et al.,2021),一个跨模态模型预训练的400 M图像+字幕对从网络上,可以用于鲁棒的自动评估图像描述,而不需要参考。跨越几个语料库的实验表明,我们的新的无参考指标,CLIPScore,实现了最高的相关性与人类的判断,优于现有的基于参考的指标,如CIDER和SPICE。信息增益实验表明,CLIPScore,其紧密关注的图像-文本兼容性,是补充现有的基于参考的指标,强调文本-文本的相似性。因此,我们还提出了一个参考增强版本,RefCLIPScore,它实现了更高的相关性。除了文字描述任务之外,几个案例研究揭示了CLIPScore表现良好的领域(剪贴画图像,替代文本评级),但与基于参考的指标相比,它也相对较弱,例如,新闻标题需要更丰富的上下文知识。
Linear Alignment of Vision-language Models for Image Captioning
ELLIS Unit Linz(ELLIS林茨单元);LIT AI Lab(LIT人工智能实验室);Institute for Machine Learning(机器学习研究所);JKU LIT SAL(约翰内斯·开普勒大学林茨软件应用实验室);Johannes Kepler University(约翰内斯·开普勒大学)
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)