GroundCap：一个视觉定位图像描述数据集
GroundCap: A Visually Grounded Image Captioning Dataset
AI总结：
提出基于ID的视觉定位图像描述系统及数据集GroundCap，实现对象引用跟踪与动作-对象链接，并引入gMETEOR指标验证描述的可验证性。
摘要：
当前的图像描述系统缺乏将描述性文本与特定视觉元素关联的能力，导致其输出难以验证。尽管近期方法提供了一些定位能力，但它们无法在多次引用间跟踪对象身份，也无法同时对动作和对象进行定位。我们提出了一种基于ID的定位系统，能够实现一致的对象引用跟踪和动作-对象链接。我们提出了GroundCap，这是一个包含来自77部电影的52,016张图像的数据集，附带344条人工标注和52,016条自动生成的描述。每条描述基于检测到的对象（132类）和动作（51类）进行定位，使用一个标签系统在将动作链接到对应对象的同时维持对象身份。我们的方法具有用于引用跟踪的持久对象ID、显式的动作-对象链接，以及通过K-means聚类对背景元素进行分割。我们提出了gMETEOR，一个将描述质量与定位准确率结合的指标，并通过在GroundCap上微调Pixtral-12B和Qwen2.5-VL 7B建立了基线性能。人工评估表明，我们的方法在生成具有连贯对象引用的可验证描述方面是有效的。
英文摘要：
Current image captioning systems lack the ability to link descriptive text to specific visual elements, making their outputs difficult to verify. While recent approaches offer some grounding capabilities, they cannot track object identities across multiple references or ground both actions and objects simultaneously. We propose a novel ID-based grounding system that enables consistent object reference tracking and action-object linking. We present GroundCap, a dataset containing 52,016 images from 77 movies, with 344 human-annotated and 52,016 automatically generated captions. Each caption is grounded on detected objects (132 classes) and actions (51 classes) using a tag system that maintains object identity while linking actions to the corresponding objects. Our approach features persistent object IDs for reference tracking, explicit action-object linking, and the segmentation of background elements through K-means clustering. We propose gMETEOR, a metric combining caption quality with grounding accuracy, and establish baseline performance by fine-tuning Pixtral-12B and Qwen2.5-VL 7B on GroundCap. Human evaluation demonstrates our approach's effectiveness in producing verifiable descriptions with coherent object references.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
