arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2205.05155cs.CV
收藏

Few-Shot图像分类基准离现实太远:使用语义任务采样更好地重建

Few-Shot Image Classification Benchmarks are Too Far From Reality: Build Back Better with Semantic Task Sampling

CentraleSupélec(中央理工-高等电力学院);Université Paris-Saclay(巴黎萨克雷大学);Sicara

Etienne Bennequin, Myriam Tami, Antoine Toubhans, Celine Hudelot

摘要:

每天,都有新的方法发布来解决Few-Shot图像分类,在学术基准上显示出越来越好的性能。然而,我们观察到这些当前的基准并不能准确地代表我们遇到的真实工业用例。在这项工作中,通过定性和定量研究,我们发现广泛使用的基准tieredImageNet强烈倾向于由语义非常不同的类组成的任务,例如浴缸,卷心菜,比萨饼,schipperke和cardoon。这使得tieredImageNet(和类似的基准测试)与评估模型解决实际用例的能力无关,这些用例通常涉及更细粒度的分类。我们使用tieredImageNet类的语义信息来减轻这种偏见,并生成一个改进的,平衡的基准。更进一步,我们还使用丹麦真菌2020数据集为Few-Shot图像分类引入了一个新的基准。该基准提出了各种各样的评估任务,具有各种细粒度。此外,该基准包括多路任务(例如,由100个类组成),这是一个具有挑战性的设置,但在工业应用中非常常见。我们的实验提出了任务的难度和其类之间的语义相似性之间的相关性,以及最先进的方法对多路Few-Shot分类的性能下降,提出了这些方法的缩放能力的问题。我们希望我们的工作将鼓励社区进一步质疑标准评估过程的质量及其与实际应用的相关性。

英文摘要:

Every day, a new method is published to tackle Few-Shot Image Classification, showing better and better performances on academic benchmarks. Nevertheless, we observe that these current benchmarks do not accurately represent the real industrial use cases that we encountered. In this work, through both qualitative and quantitative studies, we expose that the widely used benchmark tieredImageNet is strongly biased towards tasks composed of very semantically dissimilar classes e.g. bathtub, cabbage, pizza, schipperke, and cardoon. This makes tieredImageNet (and similar benchmarks) irrelevant to evaluate the ability of a model to solve real-life use cases usually involving more fine-grained classification. We mitigate this bias using semantic information about the classes of tieredImageNet and generate an improved, balanced benchmark. Going further, we also introduce a new benchmark for Few-Shot Image Classification using the Danish Fungi 2020 dataset. This benchmark proposes a wide variety of evaluation tasks with various fine-graininess. Moreover, this benchmark includes many-way tasks (e.g. composed of 100 classes), which is a challenging setting yet very common in industrial applications. Our experiments bring out the correlation between the difficulty of a task and the semantic similarity between its classes, as well as a heavy performance drop of state-of-the-art methods on many-way few-shot classification, raising questions about the scaling abilities of these methods. We hope that our work will encourage the community to further question the quality of standard evaluation processes and their relevance to real-life applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑