通过验证式原子置信度估计驯服目标幻觉
Taming Object Hallucinations with Verified Atomic Confidence Estimation
AI总结：
针对多模态大语言模型的目标幻觉问题，提出TACO框架，通过原子查询分解、自验证和置信度校准（无需外部视觉专家），在五个基准上提升模型忠实性，优于现有方法。
摘要：
多模态大语言模型（MLLMs）常受幻觉问题困扰，尤其在目标存在性、属性或关系上的错误，这削弱了其可靠性。我们提出TACO（验证式原子置信度估计），一个无需依赖外部视觉专家、通过自验证和置信度校准缓解幻觉的简单框架。TACO将响应分解为原子查询，通过释义降低对措辞的敏感性，使用自一致性（黑盒）或自置信度（灰盒）聚合估计置信度，再用语言模型优化答案。在五个基准（POPE、MME、HallusionBench、AMBER和MM-Hal Bench）上对两个MLLM（LLaVA-1.5-7B和CogVLM2）的实验显示，TACO持续优于直接提示和视觉对比解码，减少了系统偏差并改善了置信度校准，证明其在提升MLLMs忠实性方面的有效性。
英文摘要：
Multimodal Large Language Models (MLLMs) often suffer from hallucinations, particularly errors in object existence, attributes, or relations, which undermine their reliability. We introduce TACO (Verified Atomic Confidence Estimation), a simple framework that mitigates hallucinations through self-verification and confidence calibration without relying on external vision experts. TACO decomposes responses into atomic queries, paraphrases them to reduce sensitivity to wording, and estimates confidence using self-consistency (black-box) or self-confidence (gray-box) aggregation, before refining answers with a language model. Experiments on five benchmarks (POPE, MME, HallusionBench, AMBER, and MM-Hal Bench) with two MLLMs (\texttt{LLaVA-1.5-7B} and \texttt{CogVLM2}) show that TACO consistently outperforms direct prompting and Visual Contrastive Decoding, reduces systematic biases, and improves confidence calibration, demonstrating its effectiveness in enhancing the faithfulness of MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
