arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.14895cs.CVcs.AIcs.CL

通过多频扰动缓解多模态大语言模型中的物体幻觉问题

Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbations

Fudan University(复旦大学);Hikvision Research Institute(海康威视研究院)

Shuo Li, Jiajun Sun, Guodong Zheng, Xiaoran Fan, Yujiong Shen, Yi Lu, Zhiheng Xi, Yuming Yang, Wenming Tan, Tao Ji, Tao Gui, Qi Zhang, Xuanjing Huang

AI总结:

针对多模态大语言模型的物体幻觉问题,提出可插拔的多频扰动方法,通过扰动视觉特征、抑制冗余频域特征缓解幻觉,可结合推理方法在CHAIR基准上取得最优性能。

摘要:

近年来,多模态大语言模型(MLLMs)在视觉-语言任务中展现出卓越性能。然而,MLLMs生成响应的真实性常受物体幻觉问题损害。我们发现,这类幻觉的一个关键成因是模型在检测物体时过度易受特定图像频率特征的影响。本文提出多频扰动(MFP)——一种简单、低成本且可插拔的方法,该方法利用图像的低频和高频特征扰动视觉特征表示,在推理过程中显式抑制冗余的频域特征,从而缓解幻觉问题。实验结果表明,该方法在多种模型架构上均能显著缓解物体幻觉。此外,作为一种训练阶段方法,MFP可与推理阶段方法结合,在CHAIR基准上达到当前最优性能。

英文摘要:

Recently, multimodal large language models (MLLMs) have demonstrated remarkable performance in visual-language tasks. However, the authenticity of the responses generated by MLLMs is often compromised by object hallucinations. We identify that a key cause of these hallucinations is the model's over-susceptibility to specific image frequency features in detecting objects. In this paper, we introduce Multi-Frequency Perturbations (MFP), a simple, cost-effective, and pluggable method that leverages both low-frequency and high-frequency features of images to perturb visual feature representations and explicitly suppress redundant frequency-domain features during inference, thereby mitigating hallucinations. Experimental results demonstrate that our method significantly mitigates object hallucinations across various model architectures. Furthermore, as a training-time method, MFP can be combined with inference-time methods to achieve state-of-the-art performance on the CHAIR benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑