FRAMES-VQA：视觉问答中跨多模态偏移的微调鲁棒性基准
FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering
AI总结：
提出FRAMES-VQA基准，整合十个VQA数据集，分类ID与近/远OOD，评估鲁棒微调方法，量化多模态偏移，指导开发更鲁棒的微调策略。
摘要：
视觉问答（VQA）系统在适应现实世界数据偏移时面临重大挑战，尤其是在多模态情境下。虽然鲁棒的微调策略对于在分布内（ID）和分布外（OOD）场景中保持性能至关重要，但当前的评估设置主要是单模态的或特定于某些类型的OOD，对多模态情境的复杂性提供的洞察有限。在这项工作中，我们提出了一个新的基准FRAMES-VQA（视觉问答中跨多模态偏移的微调鲁棒性），用于评估VQA任务的鲁棒微调。我们利用了十个现有的VQA基准，包括VQAv2、IV-VQA、VQA-CP、OK-VQA等，并将它们分类为ID、近OOD和远OOD数据集，涵盖单模态、多模态和对抗性分布偏移。我们首先对现有的鲁棒微调方法进行了全面比较。然后，我们通过使用从各种模型中提取的单模态和多模态嵌入计算马氏距离来量化分布偏移。此外，我们进行了广泛的分析，以探索单模态和多模态偏移之间的相互作用，以及模态对ID和OOD样本的重要性。这些分析为开发更鲁棒的微调方法以处理多模态分布偏移提供了宝贵的指导。代码可在https://github.com/chengyuehuang511/FRAMES-VQA 获取。
英文摘要：
Visual question answering (VQA) systems face significant challenges when adapting to real-world data shifts, especially in multi-modal contexts. While robust fine-tuning strategies are essential for maintaining performance across in-distribution (ID) and out-of-distribution (OOD) scenarios, current evaluation settings are primarily unimodal or particular to some types of OOD, offering limited insight into the complexities of multi-modal contexts. In this work, we propose a new benchmark FRAMES-VQA (Fine-Tuning Robustness across Multi-Modal Shifts in VQA) for evaluating robust fine-tuning for VQA tasks. We utilize ten existing VQA benchmarks, including VQAv2, IV-VQA, VQA-CP, OK-VQA and others, and categorize them into ID, near and far OOD datasets covering uni-modal, multi-modal and adversarial distribution shifts. We first conduct a comprehensive comparison of existing robust fine-tuning methods. We then quantify the distribution shifts by calculating the Mahalanobis distance using uni-modal and multi-modal embeddings extracted from various models. Further, we perform an extensive analysis to explore the interactions between uni- and multi-modal shifts as well as modality importance for ID and OOD samples. These analyses offer valuable guidance on developing more robust fine-tuning methods to handle multi-modal distribution shifts. The code is available at https://github.com/chengyuehuang511/FRAMES-VQA .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
