AFTER: 通过自适应事实引导激活编辑缓解LVLM中的对象幻觉
AFTER: Mitigating the Object Hallucination of LVLM via Adaptive Factual-Guided Activation Editing
AI总结：
AFTER通过自适应事实引导激活编辑缓解LVLM中的对象幻觉，显著降低幻觉发生率。
摘要：
大规模视觉-语言模型（LVLM）在跨模态任务中取得了显著进展。然而，由于语言偏差，LVLM容易出现对象幻觉，主要分为类别、属性和关系幻觉，严重影响可信AI应用。编辑LVLM内部激活具有以最小成本缓解幻觉的潜力。然而，先前的编辑方法忽视了事实文本语义提供的有效指导，因此难以明确缓解语言偏差。为了解决这些问题，我们提出了用于幻觉缓解的自适应事实引导视觉-文本编辑（AFTER），包括事实增强激活引导（FAS）和查询自适应偏移优化（QAO），以自适应地引导原始偏置激活向事实语义。具体而言，FAS被提出为激活编辑提供事实和通用指导，从而显式建模精确的视觉-文本关联。随后，QAO引入一个查询感知的偏移估计器，从通用引导向量建立查询特定的编辑，增强编辑的多样性和粒度。在三个广泛采用的LVLM上标准幻觉基准的大量实验验证了所提AFTER的有效性，特别是在AMBER基准上，相对于基线模型，幻觉减少高达16.3%。我们的代码和数据将发布以确保可重复性。
英文摘要：
Large Vision-Language Models (LVLMs) have achieved substantial progress in cross-modal tasks. However, due to language bias, LVLMs are susceptible to object hallucination, which can be primarily divided into category, attribute, and relation hallucination, significantly impeding the trustworthy AI applications. Editing the internal activations of LVLMs has shown promising effectiveness in mitigating hallucinations with minimal cost. However, previous editing approaches neglect the effective guidance offered by factual textual semantics, thereby struggling to explicitly mitigate language bias. To address these issues, we propose Adaptive Factual-guided Visual-Textual Editing for hallucination mitigation (AFTER), which comprises Factual-Augmented Activation Steering (FAS) and Query-Adaptive Offset Optimization (QAO), to adaptively guides the original biased activations towards factual semantics. Specifically, FAS is proposed to provide factual and general guidance for activation editing, thereby explicitly modeling the precise visual-textual associations. Subsequently, QAO introduces a query-aware offset estimator to establish query-specific editing from the general steering vector, enhancing the diversity and granularity of editing. Extensive experiments on standard hallucination benchmarks across three widely adopted LVLMs validate the efficacy of the proposed AFTER, notably achieving up to a 16.3% reduction of hallucination over baseline on the AMBER benchmark. Our code and data will be released for reproducibility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
