Piculet：面向多模态大语言模型的专用模型引导幻觉减少方法
Piculet: Specialized Models-Guided Hallucination Decrease for MultiModal Large Language Models
AI总结：
本文提出免训练的Piculet方法，通过多个专用模型提取图像描述并增强MLLM输入表示，从而显著减少多模态大语言模型的幻觉问题，且可通用扩展到不同MLLMs。
摘要：
多模态大语言模型（MLLMs）在弥合视觉与语言模态之间的差距方面取得了显著进展。然而，MLLMs中的幻觉问题，即生成的文本与图像内容不一致，仍然是一个重大挑战。现有解决幻觉的方法通常依赖指令微调，这需要使用特定数据重新训练模型，从而进一步增加了使用MLLMs的成本。在本文中，我们提出了一种新颖的免训练方法，名为Piculet，用于增强MLLMs的输入表示。Piculet利用多个专用模型从输入图像中提取视觉信息的描述，并将这些描述与原始图像和查询一起作为MLLM的输入。我们从定量和定性两方面评估了我们的方法，结果表明Piculet大幅减少了MLLMs的幻觉。我们的方法可以轻松扩展到不同的MLLMs，同时具有通用性。
英文摘要：
Multimodal Large Language Models (MLLMs) have made significant progress in bridging the gap between visual and language modalities. However, hallucinations in MLLMs, where the generated text does not align with image content, continue to be a major challenge. Existing methods for addressing hallucinations often rely on instruction-tuning, which requires retraining the model with specific data, which increases the cost of utilizing MLLMs further. In this paper, we introduce a novel training-free method, named Piculet, for enhancing the input representation of MLLMs. Piculet leverages multiple specialized models to extract descriptions of visual information from the input image and combine these descriptions with the original image and query as input to the MLLM. We evaluate our method both quantitively and qualitatively, and the results demonstrate that Piculet greatly decreases hallucinations of MLLMs. Our method can be easily extended to different MLLMs while being universal.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
