分解与比较一致性：通过任务分解一致性比较衡量视觉语言模型（VLM）的答案可靠性
Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparison
AI总结：
针对现有VLM答案可靠性评估方法存在过度自信、受确认偏差影响的问题，提出通过比较直接答案与分解子问题推理所得间接答案间一致性的DeCC方法，实验证明其可靠性估计与任务准确率相关性更优。
摘要：
尽管取得了巨大进展，当前最先进的视觉语言模型（Vision-Language Models, VLMs）仍远非完美，它们容易产生幻觉，还可能生成有偏见的回答。在这种情况下，拥有一种评估VLM生成的特定回答可靠性的方法十分有用。现有方法，例如利用答案似然估计不确定性或基于提示的置信度生成方法，往往存在过度自信的问题。其他方法采用自一致性比较，但会受到确认偏差的影响。为缓解这些问题，我们提出用于可靠性测量的分解与比较一致性（Decompose and Compare Consistency, DeCC）方法。该方法通过比较VLM利用内部推理过程生成的直接答案，与将问题分解为子问题并对VLM生成的子答案进行推理所得的间接答案之间的一致性，来衡量VLM直接答案的可靠性。在6项视觉语言任务上使用3种VLM开展的实验表明，与现有方法相比，DeCC的可靠性估计与任务准确率的相关性更强。
英文摘要：
Despite tremendous advancements, current state-of-the-art Vision-Language Models (VLMs) are still far from perfect. They tend to hallucinate and may generate biased responses. In such circumstances, having a way to assess the reliability of a given response generated by a VLM is quite useful. Existing methods, such as estimating uncertainty using answer likelihoods or prompt-based confidence generation, often suffer from overconfidence. Other methods use self-consistency comparison but are affected by confirmation biases. To alleviate these, we propose Decompose and Compare Consistency (DeCC) for reliability measurement. By comparing the consistency between the direct answer generated using the VLM's internal reasoning process, and the indirect answers obtained by decomposing the question into sub-questions and reasoning over the sub-answers produced by the VLM, DeCC measures the reliability of VLM's direct answer. Experiments across six vision-language tasks with three VLMs show DeCC's reliability estimation achieves better correlation with task accuracy compared to the existing methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
