RITUAL：随机图像变换作为大型视觉语言模型中的通用抗幻觉杠杆
RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in Large Vision Language Models
AI总结：
针对大型视觉语言模型中的幻觉问题，提出RITUAL方法，通过随机图像变换作为解码时的补充输入来调整概率分布，无需训练；并扩展RITUAL+利用模型自我反馈选择变换，在多个基准上显著减少幻觉。
摘要：
大型视觉语言模型（LVLMs）的最新进展彻底改变了机器基于视觉输入理解和生成文本响应的方式，然而它们经常产生“幻觉”输出，即错误解读视觉信息，这给可靠性和可信度带来了挑战。我们提出RITUAL，一种简单的解码方法，通过在解码过程中利用随机变换的图像作为补充输入来减少幻觉，无需额外训练或外部模型即可调整输出概率分布。我们的关键洞察是，随机变换使模型暴露于多样的视觉视角，使其能够纠正导致幻觉的错误解读。具体而言，当模型基于原始图像产生幻觉时，变换后的图像——在方向、尺度或颜色等方面有所改变——提供了替代视角，有助于重新校准模型的预测。通过整合原始图像和变换图像的概率分布，RITUAL有效减少了幻觉。为了进一步提高可靠性并解决任意变换可能带来的不稳定性，我们引入了RITUAL+，这是一种扩展方法，基于LVLM的自我反馈选择图像变换。RITUAL+不是随机应用变换，而是利用LVLM评估并选择在给定上下文中对减少幻觉最有益的变换。这种自适应方法减轻了某些变换对特定任务的潜在负面影响，确保在不同场景下性能更加一致。实验表明，RITUAL和RITUAL+在多个对象幻觉基准上显著减少了幻觉。
英文摘要：
Recent advancements in Large Vision Language Models (LVLMs) have revolutionized how machines understand and generate textual responses based on visual inputs, yet they often produce "hallucinatory" outputs that misinterpret visual information, posing challenges in reliability and trustworthiness. We propose RITUAL, a simple decoding method that reduces hallucinations by leveraging randomly transformed images as complementary inputs during decoding, adjusting the output probability distribution without additional training or external models. Our key insight is that random transformations expose the model to diverse visual perspectives, enabling it to correct misinterpretations that lead to hallucinations. Specifically, when a model hallucinates based on the original image, the transformed images -- altered in aspects such as orientation, scale, or color -- provide alternative viewpoints that help recalibrate the model's predictions. By integrating the probability distributions from both the original and transformed images, RITUAL effectively reduces hallucinations. To further improve reliability and address potential instability from arbitrary transformations, we introduce RITUAL+, an extension that selects image transformations based on self-feedback from the LVLM. Instead of applying transformations randomly, RITUAL+ uses the LVLM to evaluate and choose transformations that are most beneficial for reducing hallucinations in a given context. This self-adaptive approach mitigates the potential negative impact of certain transformations on specific tasks, ensuring more consistent performance across different scenarios. Experiments demonstrate that RITUAL and RITUAL+ significantly reduce hallucinations across several object hallucination benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
