选择性“选择性预测”：减少视觉-语言推理中不必要的弃权（不执行）
Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning
AI总结：
针对选择性视觉-语言系统过度弃权（不执行）的问题，本文提出推理时算法 ReCoVERR，通过 LLM 提问并收集图像证据挽救低置信度正确预测，在不降低准确率的情况下多回答至多 20% 的问题。
摘要：
选择性预测通过允许视觉-语言模型（VLM）在不确定时弃权（不执行）回答，来最大限度减少其错误预测。然而，在部署对不准确预测容忍度较低的视觉-语言系统时，选择性预测可能会过于谨慎，并过于频繁地弃权（不执行），甚至在许多正确预测上也是如此。我们提出 ReCoVERR，这是一种推理时算法，用于减少选择性视觉-语言系统的过度弃权（不执行），同时不提高系统预测的错误率。当 VLM 作出低置信度预测时，ReCoVERR 不会弃权（不执行），而是尝试在图像中寻找相关线索，为该预测提供额外证据。ReCoVERR 使用 LLM 向 VLM 提出相关问题，收集高置信度证据；如果有足够证据证实该预测，系统就作出预测而不是弃权（不执行）。ReCoVERR 使三个 VLM（BLIP2、InstructBLIP 和 LLaVA-1.5）在 VQAv2 和 A-OKVQA 任务上能够多回答至多 20% 的问题，同时不降低系统准确率，从而提升整体系统可靠性。我们的代码可在 https://github.com/tejas1995/ReCoVERR 获取。
英文摘要：
Selective prediction minimizes incorrect predictions from vision-language models (VLMs) by allowing them to abstain from answering when uncertain. However, when deploying a vision-language system with low tolerance for inaccurate predictions, selective prediction may be over-cautious and abstain too frequently, even on many correct predictions. We introduce ReCoVERR, an inference-time algorithm to reduce the over-abstention of a selective vision-language system without increasing the error rate of the system's predictions. When the VLM makes a low-confidence prediction, instead of abstaining ReCoVERR tries to find relevant clues in the image that provide additional evidence for the prediction. ReCoVERR uses an LLM to pose related questions to the VLM, collects high-confidence evidences, and if enough evidence confirms the prediction the system makes a prediction instead of abstaining. ReCoVERR enables three VLMs (BLIP2, InstructBLIP, and LLaVA-1.5) to answer up to 20% more questions on the VQAv2 and A-OKVQA tasks without decreasing system accuracy, thus improving overall system reliability. Our code is available at https://github.com/tejas1995/ReCoVERR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
