Magnifier Prompt：通过极简指令应对多模态幻觉
Magnifier Prompt: Tackling Multimodal Hallucination via Extremely Simple Instructions
AI总结：
针对多模态大语言模型的幻觉问题，提出无需训练的Magnifier Prompt方法，基于两项核心原则设计极简指令，适用于开源闭源模型，效果媲美甚至优于复杂方法，为相关研究提供见解。
摘要：
多模态大语言模型（MLLMs）的幻觉问题阻碍了其实际应用。为解决这一问题，我们提出了Magnifier Prompt（MagPrompt，放大镜提示），这是一种通过极简指令应对MLLMs幻觉的简单却有效的方法。MagPrompt基于以下两项指导多种有效提示设计、展现出鲁棒性的核心原则：（1）MLLMs应更多关注图像；（2）当图像与模型内部知识存在冲突时，MLLMs应优先考虑图像。MagPrompt无需训练，可应用于开源与闭源模型，例如GPT-4o和Gemini-pro。它在多个数据集上表现良好，效果与VCD等更复杂的方法相当甚至更优。此外，我们的提示设计原则与实验分析为多模态幻觉研究提供了宝贵见解。
英文摘要：
Hallucinations in multimodal large language models (MLLMs) hinder their practical applications. To address this, we propose a Magnifier Prompt (MagPrompt), a simple yet effective method to tackle hallucinations in MLLMs via extremely simple instructions. MagPrompt is based on the following two key principles, which guide the design of various effective prompts, demonstrating robustness: (1) MLLMs should focus more on the image. (2) When there are conflicts between the image and the model's inner knowledge, MLLMs should prioritize the image. MagPrompt is training-free and can be applied to open-source and closed-source models, such as GPT-4o and Gemini-pro. It performs well across many datasets and its effectiveness is comparable or even better than more complex methods like VCD. Furthermore, our prompt design principles and experimental analyses provide valuable insights into multimodal hallucination.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
