Self-Aug: 用于大视觉-语言模型的查询和熵自适应解码
Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Models
AI总结：
本研究提出Self-Aug，一种无需训练的解码策略，通过自增强提示和自适应阈值算法提升大视觉-语言模型的事实一致性。
摘要：
大视觉-语言模型（LVLMs）展现了显著的多模态能力，但它们继承了底层语言模型的幻觉倾向。虽然已提出视觉对比解码来缓解这一问题，但现有方法通常应用通用的视觉增强，忽视了文本查询提供的特定上下文，限制了其有效性。本研究介绍了一种无需训练的解码策略，解决了这些限制，具有两个关键贡献。首先，一种自增强提示策略，利用模型的内在知识，动态对齐查询与视觉增强之间的语义。其次，一种自适应阈值算法，根据输出稀疏性自适应调整下一个token候选大小，利用logit分布的全部信息。在四个LVLM和七个基准测试中进行的广泛实验表明，所提出的解码显著提高了事实一致性，优于最先进的解码方法。本工作强调了整合查询依赖增强和熵感知解码对于提高LVLM有效生成的重要性。
英文摘要：
Large Vision-Language Models (LVLMs) have demonstrated remarkable multimodal capabilities, but they inherit the tendency to hallucinate from their underlying language models. While visual contrastive decoding has been proposed to mitigate this issue, existing methods often apply generic visual augmentations that disregard the specific context provided by the text query, limiting their effectiveness. This study introduces a novel training-free decoding strategy that addresses these limitations, featuring two key contributions. First, a self-augmentation prompting strategy that leverages the intrinsic knowledge of the model to dynamically align semantics between the query and the visual augmentation. Second, an adaptive thresholding algorithm that adaptively adjusts next token candidate size based on the output sparsity, utilizing full information from the logit distribution. Extensive experiments across four LVLMs and seven benchmarks demonstrate that the proposed decoding significantly enhances factual consistency compared to state-of-the-art decoding methods. This work highlights the importance of integrating query-dependent augmentation and entropy-aware decoding for improving effective generation of LVLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
