尽管视觉问答（Visual Question Answering, VQA）领域已取得诸多进展，但模型评估自身答案正确性的能力仍未得到充分研究。近期研究表明，现成的VQA模型在答错时很难做到弃权（不执行）作答。弃权选项也被称为选择性预测，在向必须信任系统输出的用户（例如为视障用户提供服务的VQA助手）部署系统时极具实用价值。在这类场景中，弃权功能尤为重要，因为用户可能提供分布外（out-of-distribution, OOD）输入或对抗性输入，导致模型给出错误答案的概率升高。\n在本研究中，我们在分布内（in-distribution, ID）和分布外两种场景下探索选择性VQA任务，实验中模型会接收分布内与分布外混合的数据。该任务的目标是在最小化已答题目错误风险的前提下，尽可能提升可回答的问题数量。我们提出了一种简单却有效的“向同伴学习（Learning from Your Peers, LYP）”方法，用于训练多模态选择函数以做出弃权决策。该方法利用在训练数据不同子集上训练得到的模型的预测结果，作为优化选择性VQA模型的目标。它无需额外人工标注或留出数据，还能提供信号，用于识别容易/难以泛化的样本。\n在大量评估实验中，我们证明该方法能为不同架构、不同规模的多种模型带来性能提升。总体而言，在分布内场景下，我们的方法在1%错误风险下的选择性预测指标覆盖率（C@1%）达到32.92%，是该任务此前最佳覆盖率15.79%的两倍。在分布内/分布外混合场景下，利用模型的softmax置信度做弃权决策的效果极差，即便仅面对10%的分布外样本，在1%错误风险下能回答的问题占比还不到5%，但使用LYP方法训练得到的选择函数可将该指标提升至25.38% C@1%。
英文摘要：
Despite advances in Visual Question Answering (VQA), the ability of models to assess their own correctness remains underexplored. Recent work has shown that VQA models, out-of-the-box, can have difficulties abstaining from answering when they are wrong. The option to abstain, also called Selective Prediction, is highly relevant when deploying systems to users who must trust the system's output (e.g., VQA assistants for users with visual impairments). For such scenarios, abstention can be especially important as users may provide out-of-distribution (OOD) or adversarial inputs that make incorrect answers more likely. In this work, we explore Selective VQA in both in-distribution (ID) and OOD scenarios, where models are presented with mixtures of ID and OOD data. The goal is to maximize the number of questions answered while minimizing the risk of error on those questions. We propose a simple yet effective Learning from Your Peers (LYP) approach for training multimodal selection functions for making abstention decisions. Our approach uses predictions from models trained on distinct subsets of the training data as targets for optimizing a Selective VQA model. It does not require additional manual labels or held-out data and provides a signal for identifying examples that are easy/difficult to generalize to. In our extensive evaluations, we show this benefits a number of models across different architectures and scales. Overall, for ID, we reach 32.92% in the selective prediction metric coverage at 1% risk of error (C@1%) which doubles the previous best coverage of 15.79% on this task. For mixed ID/OOD, using models' softmax confidences for abstention decisions performs very poorly, answering <5% of questions at 1% risk of error even when faced with only 10% OOD examples, but a learned selection function with LYP can increase that to 25.38% C@1%.
UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation
School of Electrical and Computer Engineering, Peking University（北京大学电气与计算机工程学院）；Hupan Lab（湖畔实验室）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；University of Notre Dame（圣母大学）；Tsinghua University（清华大学）
VGAT: A Cancer Survival Analysis Framework Transitioning from Generative Visual Question Answering to Genomic Reconstruction
1 Academy for Engineering；Technology, Fudan University, Shanghai, China；2 Institute of Metaverse \& Intelligent Medicine, Fudan University, Shanghai, China；3 Engineering Research Center of AI；4 Jilin Provincial Key Laboratory of Intelligence Science
From Pixels to Objects: Cubic Visual Attention for Visual Question Answering
Center for Future Media（未来媒体中心）；School of Computer Science and Engineering（计算机科学与工程学院）；University of Electronic Science and Technology of China（电子科技大学）
AV-Master: Dual-Path Comprehensive Perception Makes Better Audio-Visual Question Answering
Great Bay University（大湾区大学）；Dongguan Key Laboratory for Intelligence and Information Technology（东莞智能与信息科技重点实验室）；Nankai University（南开大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；The Chinese University of Hong Kong（香港中文大学）
A Knowledge Noise Mitigation Framework for Knowledge-based Visual Question Answering
School of Computer, Electronics and Information（计算机、电子与信息学院）；Guangxi University（广西大学）；Guangxi Key Laboratory of Multimedia Communications and Network Technology（广西多媒体通信与网络技术重点实验室）
SeVeR: Selective Visual Exposure and Retrieval for 3D Medical Image Question Answering
DAMO Academy, Alibaba Group（阿里巴巴达摩院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Hupan Lab（湖畔实验室）；School of Public Health, Zhejiang University（浙江大学公共卫生学院）
Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answering
LTCI, Télécom-Paris, Institut Polytechnique de Paris（LTCI， Télécom-Paris， Institut Polytechnique de Paris）；Inria, LJK, Univ. Grenoble Alpes（Inria， LJK， 火车头 Grenoble Alpes）；Universitat Autónoma de Barcelona（Autonomous University of Barcelona）
WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室 (MAIS)）
TAP-RAG: Task-Aware Policy Control for Long-Document Multimodal Question Answering
School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；The International Joint Institute of Tianjin University（天津大学国际联合学院）；Department of Automation, Tsinghua University（清华大学自动化系）
TemporalDoRA: Temporal PEFT for Robust Surgical Video Question Answering
Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB), Politecnico di Milano（电子信息与生物工程系（DEIB），米兰理工学院）；IRCCS Humanitas Research Hospital（人类学研究所医院）；UCL Hawkes Institute and Department of Computer Science, University College London（伦敦大学学院 Hawkes 研究所和计算机科学系）；Division of Informatics, Imaging and Data Science, University of Manchester（信息学、影像学与数据科学系，曼彻斯特大学）
