arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.03147cs.CVcs.AI
收藏

通过鲁棒性分析与基础问题链的上下文学习改进视觉问答模型

Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questions

University of Amsterdam(阿姆斯特丹大学);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学)

Jia-Hong Huang, Modar Alfadly, Bernard Ghanem, Marcel Worring

AI总结:

本研究提出利用基础问题作为噪声评估VQA模型鲁棒性的新方法,通过LASSO优化排序生成合理噪声水平,并引入R_score指标和数据集,同时发现基础问题链的上下文学习能提升模型准确率。

摘要:

深度神经网络在视觉问答(VQA)任务中一直发挥着关键作用,传统研究主要集中于提升模型准确率。然而,近期出现了一种评估这些模型对抗对抗性攻击鲁棒性的趋势。这涉及在输入噪声程度不断增加的情况下评估VQA模型的准确率,这些噪声可以针对图像或所提出的查询问题,即所谓的主问题。然而,目前缺乏对VQA这一方面的适当分析。本研究提出了一种新方法,利用与主问题在语义上相关的问题(称为基础问题)作为噪声来评估VQA模型的鲁棒性。研究假设,随着基础问题与主问题相似度的降低,噪声水平会增加。为了为给定的主问题生成合理的噪声水平,根据基础问题与主问题的相似度对基础问题池进行排序,并将该排序问题转化为一个LASSO优化问题。此外,本研究还提出了一种新的鲁棒性度量指标R_score,以及两个基础问题数据集,以标准化VQA模型鲁棒性分析。实验结果表明,所提出的评估方法能够有效分析VQA模型的鲁棒性。此外,实验还表明,利用基础问题链进行上下文学习可以提升模型准确率。

英文摘要:

Deep neural networks have been critical in the task of Visual Question Answering (VQA), with research traditionally focused on improving model accuracy. Recently, however, there has been a trend towards evaluating the robustness of these models against adversarial attacks. This involves assessing the accuracy of VQA models under increasing levels of noise in the input, which can target either the image or the proposed query question, dubbed the main question. However, there is currently a lack of proper analysis of this aspect of VQA. This work proposes a new method that utilizes semantically related questions, referred to as basic questions, acting as noise to evaluate the robustness of VQA models. It is hypothesized that as the similarity of a basic question to the main question decreases, the level of noise increases. To generate a reasonable noise level for a given main question, a pool of basic questions is ranked based on their similarity to the main question, and this ranking problem is cast as a LASSO optimization problem. Additionally, this work proposes a novel robustness measure, R_score, and two basic question datasets to standardize the analysis of VQA model robustness. The experimental results demonstrate that the proposed evaluation method effectively analyzes the robustness of VQA models. Moreover, the experiments show that in-context learning with a chain of basic questions can enhance model accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑