EFUF：用于缓解多模态大语言模型幻觉的高效细粒度遗忘框架
EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models
AI总结：
提出高效细粒度遗忘框架EFUF，无需配对数据即可消除多模态大语言模型的物体幻觉，在降低幻觉的同时保持生成质量且计算开销适中。
摘要：
多模态大语言模型（MLLMs）在过去几年中引起了越来越多的关注，但它们仍可能生成包含相应图像中不存在的物体的描述，这种现象被称为物体幻觉。为了消除幻觉，现有方法手动标注有无幻觉的配对响应，随后采用各种对齐算法来提升图像与文本之间的对齐能力。然而，这些方法不仅在微调阶段需要大量计算资源，还需要昂贵的人工标注来构建对齐算法所需的配对数据。为了解决这些问题，我们借鉴遗忘的思想，提出了一种高效细粒度遗忘框架（EFUF），该框架无需配对数据即可消除幻觉。大量实验表明，我们的方法在保持生成质量的同时，持续降低幻觉，且计算开销适中。我们的代码和数据集将公开提供。
英文摘要：
Multimodal large language models (MLLMs) have attracted increasing attention in the past few years, but they may still generate descriptions that include objects not present in the corresponding images, a phenomenon known as object hallucination. To eliminate hallucinations, existing methods manually annotate paired responses with and without hallucinations, and then employ various alignment algorithms to improve the alignment capability between images and text. However, they not only demand considerable computation resources during the finetuning stage but also require expensive human annotation to construct paired data needed by the alignment algorithms. To address these issues, we borrow the idea of unlearning and propose an efficient fine-grained unlearning framework (EFUF), which can eliminate hallucinations without the need for paired data. Extensive experiments show that our method consistently reduces hallucinations while preserving the generation quality with modest computational overhead. Our code and datasets will be publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
