arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2004.05595cs.CV
收藏

基于熵聚类的视觉问题难度评估方法

An Entropy Clustering Approach for Assessing Visual Question Difficulty

Hiroshima University(广岛大学);National Institute of Informatics(国立信息学研究所)

Kento Terao, Toru Tamaki, Bisser Raytchev, Kazufumi Kaneda, Shun'ichi Satoh

摘要:

我们提出了一种新的方法来识别视觉问题的难度视觉问答(VQA)没有直接的监督或注释的困难。先前的工作已经考虑了人类注释者的地面实况答案的多样性。相比之下,我们分析了基于多个不同VQA模型的行为的视觉问题的难度。我们建议对由三种不同模型获得的预测答案分布的熵值进行聚类:一种是将图像和问题作为输入的基线方法,另一种是仅将图像和问题作为输入的两种变体。我们使用一个简单的k-means来聚类VQA v2验证集的视觉问题。然后,我们使用最先进的方法来确定每个聚类的答案分布的准确性和熵。所提出的方法的一个好处是,没有注释的难度是必需的,因为每个集群的准确性反映了视觉问题的难度属于it. We的方法可以识别集群的困难视觉问题,没有正确回答的国家的最先进的方法。对VQA v2数据集的分析表明:1)所有方法在最困难的聚类(约10%的准确率)上表现不佳; 2)随着聚类难度的增加,不同方法预测的答案开始不同; 3)聚类熵的值与聚类准确率高度相关。我们表明,我们的方法具有的优势,能够评估视觉问题的难度,没有地面的真相(\ie,VQA v2的测试集),通过将它们分配到一个集群。我们希望这可以刺激新的研究方向和新算法的发展。

英文摘要:

We propose a novel approach to identify the difficulty of visual questions for Visual Question Answering (VQA) without direct supervision or annotations to the difficulty. Prior works have considered the diversity of ground-truth answers of human annotators. In contrast, we analyze the difficulty of visual questions based on the behavior of multiple different VQA models. We propose to cluster the entropy values of the predicted answer distributions obtained by three different models: a baseline method that takes as input images and questions, and two variants that take as input images only and questions only. We use a simple k-means to cluster the visual questions of the VQA v2 validation set. Then we use state-of-the-art methods to determine the accuracy and the entropy of the answer distributions for each cluster. A benefit of the proposed method is that no annotation of the difficulty is required, because the accuracy of each cluster reflects the difficulty of visual questions that belong to it. Our approach can identify clusters of difficult visual questions that are not answered correctly by state-of-the-art methods. Detailed analysis on the VQA v2 dataset reveals that 1) all methods show poor performances on the most difficult cluster (about 10\% accuracy), 2) as the cluster difficulty increases, the answers predicted by the different methods begin to differ, and 3) the values of cluster entropy are highly correlated with the cluster accuracy. We show that our approach has the advantage of being able to assess the difficulty of visual questions without ground-truth (\ie, the test set of VQA v2) by assigning them to one of the clusters. We expect that this can stimulate the development of novel directions of research and new algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑