变分视觉问答用于不确定性感知的选择性预测
Variational Visual Question Answering for Uncertainty-Aware Selective Prediction
AI总结：
本文提出变分VQA方法，通过变分贝叶斯方法提升视觉问答和视觉推理任务的可靠性，尤其在低误差容忍度下表现优异，且优于AdamW训练模型。
摘要：
尽管近年来在视觉语言模型（VLMs）方面取得了显著进展，但在视觉问答（VQA）和视觉推理等任务中，仍容易表现出过度自信和幻觉的问题。贝叶斯方法可以通过帮助模型选择性地响应来提高可靠性，即模型只在足够自信时作出回应。然而，此类方法对于大型模型来说成本高昂且效果不佳，且在多模态应用中缺乏证据支持。本文首次展示了变分贝叶斯方法在VQA中的选择性预测的有效性和竞争力。我们基于深度学习变分方法的最新进展，提出了一种称为“变分VQA”的扩展方法。该方法提高了校准性，并在VQA和视觉推理中的选择性预测上取得了显著提升，特别是在误差容忍度较低（≤1%）的情况下。通常，只需一个后验样本就能产生比使用AdamW训练的模型更可靠的答案。此外，我们还提出了一种新的风险规避选择器，通过考虑预测的方差来超越标准样本平均。总体而言，我们提供了有力的证据表明，变分学习是使大型VLMs更安全、更可信的一种可行方法。
英文摘要：
Despite remarkable progress in recent years, Vision Language Models (VLMs) remain prone to overconfidence and hallucinations on tasks such as Visual Question Answering (VQA) and Visual Reasoning. Bayesian methods can potentially improve reliability by helping models predict selectively, that is, models respond only when they are sufficiently confident. Unfortunately, such approaches can be costly and ineffective for large models, and there exists little evidence to show otherwise for multimodal applications. Here, we show for the first time the effectiveness and competitive edge of variational Bayes for selective prediction in VQA. We build on recent advances in variational methods for deep learning and propose an extension called "Variational VQA". This method improves calibration and yields significant gains for selective prediction on VQA and Visual Reasoning, particularly when the error tolerance is low ($\leq 1\%$). Often, just one posterior sample yields more reliable answers than those given by models trained with AdamW. In addition, we propose a new risk-averse selector that outperforms standard sample averaging by considering the variance of predictions. Overall, we present compelling evidence that variational learning is a viable option to make large VLMs safer and more trustworthy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
