单次优化的转向向量用于VLMs的幻觉缓解
One-shot Optimized Steering Vector for Hallucination Mitigation for VLMs
AI总结：
本文提出OSGA框架，通过单次优化生成通用转向向量，有效缓解VLMs的幻觉问题并提升安全性。
摘要：
视觉语言模型（VLMs）在多模态任务上实现了强大的性能，但仍然存在幻觉和安全相关故障，即使在大规模情况下也持续存在。转向提供了一种轻量的技术来提高模型性能。然而，无论转向是输入依赖还是输入独立，都在效率和有效性之间取得有意义的权衡。在本工作中，我们观察到转向向量可以在任务共享对齐语义意图时跨输入泛化。基于这一见解，我们提出了OSGA（单次转向与生成锚点），一种输入独立的框架，通过单次优化实例提高模型性能。OSGA首先通过基于方差的数据选择策略选择一个信息丰富的样本，并通过对比目标与生成锚点正则化学习一个单个转向向量。所得向量可以在推理过程中在特定层上普遍应用，而无需修改模型参数。在多个基准上的实验表明，一个单次OSGA优化的转向向量一致地提高了幻觉缓解和安全增强，且开销很小，突显了单次转向作为可靠VLMs的实用且可扩展的解决方案。
英文摘要：
Vision Language Models (VLMs) achieve strong performance on multimodal tasks but still suffer from hallucination and safety-related failures that persist even at scale. Steering offers a lightweight technique to improve model performance. However, steering, whether input-dependent or input-independent, achieves a meaningful trade-off between efficiency and effectiveness. In this work, we observe that steering vectors can generalize across inputs when tasks share aligned semantic intent. Based on this insight, we propose \textbf{OSGA} (\textbf{O}ne-shot \textbf{S}teering with \textbf{G}enerative \textbf{A}nchor), an input-independent framework that improves model performance with a single optimization instance. OSGA first selects an informative sample via a variance-based data selection strategy and learns a single steering vector with a contrastive objective with generative anchor regularization. The resulting vector can be universally applied at a certain layer during inference time without modifying model parameters. Experiments across multiple benchmarks show that a single OSGA-optimized steering vector consistently improves hallucination mitigation and safety enhancement with negligible overhead, highlighting one-shot steering as a practical and scalable solution for reliable VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
