跨模态身份映射：通过强化学习最小化模态转换中的信息损失
Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning
AI总结：
提出跨模态身份映射（CIM）框架，利用强化学习优化图像描述，通过检索一致性度量信息损失，无需额外标注，显著提升关系推理能力。
摘要：
大型视觉语言模型（LVLMs）在生成的图像描述中常常遗漏或歪曲关键的视觉内容。最小化这种信息损失将迫使LVLMs关注图像细节以生成精确的描述。然而，由于视觉内容和文本输出之间的模态差距，衡量模态转换过程中的信息损失本质上是困难的。在本文中，我们认为图像描述的质量与使用该描述通过文本搜索检索到的图像之间的相似性正相关。基于这一见解，我们进一步提出了跨模态身份映射（CIM），一种无需额外标注即可增强图像描述的强化学习框架。具体来说，该方法从两个角度定量评估信息损失：图库表示一致性和查询-图库图像相关性。在这些指标的监督下，LVLM最小化信息损失并旨在实现从图像到描述的恒等映射。实验结果表明，我们的方法在图像描述方面表现出优越的性能，即使与监督微调相比也是如此。特别是在COCO-LN500基准上，CIM在Qwen2.5-VL-7B上的关系推理提升了20%。
英文摘要：
Large Vision-Language Models (LVLMs) often omit or misrepresent critical visual content in generated image captions. Minimizing such information loss will force LVLMs to focus on image details to generate precise descriptions. However, measuring information loss during modality conversion is inherently challenging due to the modal gap between visual content and text output. In this paper, we argue that the quality of an image caption is positively correlated with the similarity between images retrieved via text search using that caption. Based on this insight, we further propose Cross-modal Identity Mapping (CIM), a reinforcement learning framework that enhances image captioning without requiring additional annotations. Specifically, the method quantitatively evaluates the information loss from two perspectives: Gallery Representation Consistency and Query-gallery Image Relevance. Supervised under these metrics, LVLM minimizes information loss and aims to achieve identity mapping from images to captions. The experimental results demonstrate the superior performance of our method in image captioning, even when compared with Supervised Fine-Tuning. Particularly, on the COCO-LN500 benchmark, CIM achieves a 20% improvement in relation reasoning on Qwen2.5-VL-7B.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
