arXiv 2603.182822026-03-20cs.CV

CycleCap: 通过自监督循环一致性微调提升VLMs的描述性能

CycleCap: Improving VLMs Captioning Performance via Self-Supervised Cycle Consistency Fine-Tuning

Queen Mary, University of London（伦敦大学玛丽女王学院）；Centre for Research and Technology Hellas（希腊研究中心）；City St George’s, University of London（伦敦大学圣乔治学院）

Marios Krestenitis, Christos Tzelepis, Konstantinos Ioannidis, Stefanos Vrochidis, Ioannis Kompatsiaris, Georgios Tzimiropoulos, Shaogang Gong, Ioannis Patras

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出CycleCap，通过自监督循环一致性训练提升VLMs的图像描述性能，利用GRPO优化策略，基于重建图像与原始图像相似性作为奖励信号，无需标注数据即可提高描述准确性。

摘要：

视觉语言模型（VLMs）在图像描述、视觉问答和视觉推理方面取得了显著进展，但仍然容易出现视觉-语言对齐问题，常产生过于笼统或虚构的描述。现有方法通过指令微调或复杂测试时间框架来解决，但成本高或复杂。本文从循环一致性角度重新审视图像-文本对齐：给定一张图像和由图像到文本模型生成的描述，通过文本到图像模型反向映射应重建与原始图像接近的图像。在我们的设置中，VLM作为图像到文本组件，预训练的文本到图像模型通过重建生成的描述来闭合循环。基于此，我们引入CycleCap，一种通过组相对策略优化（GRPO）改进图像描述的微调方案，奖励基于原始图像与重建图像的相似性，实时计算。不同于以往使用循环一致性损失构建偏好数据集的方法，我们的方法直接利用循环一致性作为自监督训练信号。这使能够仅使用原始图像，无需定制化的图像-文本数据集，同时引导VLM生成更准确和基于现实的文本描述。应用于从1B到7B参数的四个VLMs，CycleCap在描述和幻觉基准上均取得一致改进，超越了依赖监督循环一致性训练的最先进方法。

英文摘要：

Visual-Language Models (VLMs) have achieved remarkable progress in image captioning, visual question answering, and visual reasoning. Yet they remain prone to vision-language misalignment, often producing overly generic or hallucinated descriptions. Existing approaches address this via instruction tuning-requiring costly, large-scale annotated datasets or via complex test-time frameworks for caption refinement. In this work, we revisit image-text alignment through the lens of cycle consistency: given an image and a caption generated by an image-to-text model, the backward mapping through a text-to-image model should reconstruct an image that closely matches the original. In our setup, a VLM serves as the image-to-text component, while a pre-trained text-to-image model closes the loop by reconstructing the image from the generated caption. Building on this, we introduce CycleCap, a fine-tuning scheme to improve image captioning using Group Relative Policy Optimization (GRPO) with a reward based on the similarity between the original and reconstructed images, computed on-the-fly. Unlike previous work that uses cycle consistency loss for preference dataset construction, our method leverages cycle consistency directly as a self-supervised training signal. This enables the use of raw images alone, eliminating the need for curated image-text datasets, while steering the VLM to produce more accurate and grounded text descriptions. Applied to four VLMs ranging from 1B to 7B parameters, CycleCap yields consistent improvements across captioning and hallucination benchmarks, surpassing state-of-the-art methods that rely on supervised cycle consistency training.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。