arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.00259cs.CVcs.CL
收藏

Super-CLEVR:一个用于诊断视觉推理中领域鲁棒性的虚拟基准

Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning

Johns Hopkins University(约翰斯·霍普金斯大学);University of Southern California(南加州大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);University of Freiburg(弗莱堡大学)

Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, Alan Yuille

AI总结:

本文提出 Super-CLEVR 虚拟基准,以隔离视觉复杂性、问题冗余、概念分布与组合性四类领域偏移,并通过 P-NSVQA 证明解耦推理、感知与概率不确定性可提升 VQA 鲁棒性。

摘要:

视觉问答(VQA)模型在分布外数据上往往表现不佳,并且在领域泛化方面存在困难。由于该任务的多模态性质,多种变化因素相互交织,使得泛化难以分析。这促使我们提出一个虚拟基准 Super-CLEVR,在其中可以隔离 VQA 领域偏移中的不同因素,从而独立研究它们的影响。本文考虑四个因素:视觉复杂性、问题冗余、概念分布以及概念组合性。借助可控生成的数据,Super-CLEVR 使我们能够在测试数据沿上述每个维度不同于训练数据的情形下测试 VQA 方法。我们研究了四种现有方法,包括两种神经符号方法 NSCL 和 NSVQA,以及两种非符号方法 FiLM 和 mDETR;还研究了我们提出的方法 probabilistic NSVQA(P-NSVQA),该方法通过不确定性推理扩展 NSVQA。P-NSVQA 在四个领域偏移因素中的三个上优于其他方法。结果表明,将推理与感知解耦,并结合概率不确定性,可形成对领域偏移更鲁棒的强大 VQA 模型。数据集和代码发布于 https://github.com/Lizw14/Super-CLEVR。

英文摘要:

Visual Question Answering (VQA) models often perform poorly on out-of-distribution data and struggle on domain generalization. Due to the multi-modal nature of this task, multiple factors of variation are intertwined, making generalization difficult to analyze. This motivates us to introduce a virtual benchmark, Super-CLEVR, where different factors in VQA domain shifts can be isolated in order that their effects can be studied independently. Four factors are considered: visual complexity, question redundancy, concept distribution and concept compositionality. With controllably generated data, Super-CLEVR enables us to test VQA methods in situations where the test data differs from the training data along each of these axes. We study four existing methods, including two neural symbolic methods NSCL and NSVQA, and two non-symbolic methods FiLM and mDETR; and our proposed method, probabilistic NSVQA (P-NSVQA), which extends NSVQA with uncertainty reasoning. P-NSVQA outperforms other methods on three of the four domain shift factors. Our results suggest that disentangling reasoning and perception, combined with probabilistic uncertainty, form a strong VQA model that is more robust to domain shifts. The dataset and code are released at https://github.com/Lizw14/Super-CLEVR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑