以最小人类监督实现反馈驱动的视觉-语言对齐
Feedback-Driven Vision-Language Alignment with Minimal Human Supervision
AI总结：
针对视觉-语言模型依赖大量高质量图文训练数据的问题，提出无需人工图文对和偏好标注的SVP框架，通过少量筛选图像、自动字幕和预训练grounding模型反馈增强对齐，在多任务上实现显著性能提升。
摘要：
视觉-语言模型（VLM）在整合视觉与语言信息方面已展现出显著潜力，但其性能往往受限于对大量高质量图文训练数据的需求，而这类图文对的整理既耗时又计算成本高昂。为解决这一挑战，我们提出SVP（基于采样的视觉投影）这一新型框架，无需依赖人工整理的图文对或偏好标注即可增强视觉-语言对齐效果。SVP利用少量人工筛选的图像、自动字幕生成技术以及预训练 grounding（ grounding，即视觉 grounding，指将文本描述与图像中对应区域关联）模型作为反馈机制，挖掘VLM中的潜在信息。我们在六个核心领域评估了该方法：字幕生成、指代理解、视觉问答、多任务处理、幻觉控制和物体召回。结果显示该方法带来了显著提升：字幕生成任务平均提升14%，物体召回最高提升12%，幻觉现象大幅减少，同时保留了问答能力。借助SVP，小型VLM的幻觉减少效果可达到五倍规模更大模型的水平；而初始指代能力较弱的VLM性能提升超过一倍，接近两倍规模模型的水平。
英文摘要：
Vision-language models (VLMs) have demonstrated remarkable potential in integrating visual and linguistic information, but their performance is often constrained by the need for extensive, high-quality image-text training data. Curation of these image-text pairs is both time-consuming and computationally expensive. To address this challenge, we introduce SVP (Sampling-based Visual Projection), a novel framework that enhances vision-language alignment without relying on manually curated text-image pairs or preference annotation. SVP leverages a small set of manually selected images, self-captioning and a pre-trained grounding model as a feedback mechanism to elicit latent information in VLMs. We evaluate our approach across six key areas: captioning, referring, visual question answering, multitasking, hallucination control, and object recall. Results demonstrate significant improvements, including a 14 % average improvement in captioning tasks, up to 12 % increase in object recall, and significantly reduced hallucinations, while maintaining question-answering capabilities. Using SVP, a small VLM achieves hallucination reductions similar to a model five times larger, while a VLM with initially poor referring capabilities more than doubles its performance, approaching parity with a model twice its size.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
