arXiv 2510.271642025-10-31cs.CVcs.AI

为高分辨率图像生成准确且详细的描述

Generating Accurate and Detailed Captions for High-Resolution Images

Department of Artificial Intelligence, University of Seoul（首尔大学人工智能系）；Department of Computer Science and Engineering, POSTECH（POSTECH计算机科学与工程系）；Department of Applied Statistics, Yonsei University（延世大学应用统计系）

Hankyeol Lee, Gawon Seo, Kyounggyu Lee, Dogun Kim, Kyungwoo Song, Jiyoung Jung

备注：: Work conducted in 2024; released for archival purposes

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出结合视觉语言模型、大语言模型和目标检测系统的多阶段管线，通过识别关键物体、预测共现物体并验证，为高分辨率图像生成更详细准确的描述并有效减少幻觉。

摘要：

视觉语言模型（VLMs）通常难以生成高分辨率图像的准确且详细的描述，因为它们通常在低分辨率输入（例如 224x224 或 336x336 像素）上进行预训练。将高分辨率图像缩小到这些尺寸可能会导致视觉细节丢失和重要物体遗漏。为了解决这一限制，我们提出了一种结合视觉语言模型、大语言模型（LLMs）和目标检测系统的新型管线，以提升描述质量。该管线通过多阶段过程来优化描述。给定一张高分辨率图像，首先使用 VLM 生成初始描述，然后由 LLM 识别图像中的关键物体。LLM 预测可能与已识别关键物体共同出现的其他物体，并由目标检测系统验证这些预测。初始描述中未提及的新检测物体将进行聚焦的区域特定描述，以确保它们被纳入。此过程通过移除对未检测物体的提及来丰富描述细节，同时减少幻觉。我们使用大型多模态模型的成对比较和定量评分，以及幻觉检测基准来评估增强后的描述。在精选的高分辨率图像数据集上的实验表明，我们的管线能生成更详细、更可靠的图像描述，同时有效减少幻觉。

英文摘要：

Vision-language models (VLMs) often struggle to generate accurate and detailed captions for high-resolution images since they are typically pre-trained on low-resolution inputs (e.g., 224x224 or 336x336 pixels). Downscaling high-resolution images to these dimensions may result in the loss of visual details and the omission of important objects. To address this limitation, we propose a novel pipeline that integrates vision-language models, large language models (LLMs), and object detection systems to enhance caption quality. Our proposed pipeline refines captions through a novel, multi-stage process. Given a high-resolution image, an initial caption is first generated using a VLM, and key objects in the image are then identified by an LLM. The LLM predicts additional objects likely to co-occur with the identified key objects, and these predictions are verified by object detection systems. Newly detected objects not mentioned in the initial caption undergo focused, region-specific captioning to ensure they are incorporated. This process enriches caption detail while reducing hallucinations by removing references to undetected objects. We evaluate the enhanced captions using pairwise comparison and quantitative scoring from large multimodal models, along with a benchmark for hallucination detection. Experiments on a curated dataset of high-resolution images demonstrate that our pipeline produces more detailed and reliable image captions while effectively minimizing hallucinations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。