用人工智能增强新闻业：基于LLMs和LMMs的新闻文章上下文图像描述研究
Enhancing Journalism with AI: A Study of Contextualized Image Captioning for News Articles using LLMs and LMMs
AI总结：
本研究利用GoodNews数据集评估LLMs和LMMs为新闻图片生成上下文描述的能力，发现开源LMMs可媲美专有模型，且控制上下文量能提升性能，表明新闻业需要人在回路的交互式策略。
摘要：
大型语言模型（LLMs）和大型多模态模型（LMMs）已对人工智能社区、工业界及多个经济领域产生了显著影响。在新闻业中，整合人工智能带来了独特的挑战与机遇，尤其是在提升新闻报道的质量和效率方面。本研究探讨了LLMs和LMMs如何通过为新闻文章配图生成上下文相关的图像描述来辅助新闻实践。我们使用GoodNews数据集开展实验，以评估LMMs（BLIP-2、GPT-4v或LLaVA）整合两类上下文信息的能力：完整新闻文章或抽取出的命名实体。此外，我们将它们的性能与一个两阶段流水线进行了比较，该流水线由图像描述模型（BLIP-2、OFA或ViT-GPT2）和随后使用LLMs（GPT-4或LLaMA）进行的事后上下文化组成。我们评估了多种模型，发现尽管上下文化模型的选择对两阶段流水线而言是一个显著影响因素，但在LMMs中情况并非如此，其中较小的开源模型与专有的、由GPT驱动的模型相比表现良好。此外，我们发现控制所提供的上下文数量能够提升性能。这些结果凸显了全自动化方法的局限性，并强调了采用交互式、人在回路（human-in-the-loop）策略的必要性。
英文摘要：
Large language models (LLMs) and large multimodal models (LMMs) have significantly impacted the AI community, industry, and various economic sectors. In journalism, integrating AI poses unique challenges and opportunities, particularly in enhancing the quality and efficiency of news reporting. This study explores how LLMs and LMMs can assist journalistic practice by generating contextualised captions for images accompanying news articles. We conducted experiments using the GoodNews dataset to evaluate the ability of LMMs (BLIP-2, GPT-4v, or LLaVA) to incorporate one of two types of context: entire news articles, or extracted named entities. In addition, we compared their performance to a two-stage pipeline composed of a captioning model (BLIP-2, OFA, or ViT-GPT2) with post-hoc contextualisation with LLMs (GPT-4 or LLaMA). We assess a diversity of models, and we find that while the choice of contextualisation model is a significant factor for the two-stage pipelines, this is not the case in the LMMs, where smaller, open-source models perform well compared to proprietary, GPT-powered ones. Additionally, we found that controlling the amount of provided context enhances performance. These results highlight the limitations of a fully automated approach and underscore the necessity for an interactive, human-in-the-loop strategy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
