基准测试的基准测试:MS-COCO的可靠性如何?
Benchmarking a Benchmark: How Reliable is MS-COCO?
AI总结:
本研究针对基准数据集常被侧重结果却忽视内容偏差的问题,利用重新标注的Sama-COCO数据集和形状分析流水线挖掘MS-COCO的潜在偏差,通过跨数据集的模型训练评估证实标注风格的重要性,为标注流水线优化提供依据并公开了数据集。
摘要:
基准数据集用于分析和比较从图像分类到分割等各类任务的算法,在图像预训练算法中也发挥着重要作用。人们往往侧重结果,却很少关注数据集内的实际内容。质疑从这些数据集中能学到何种信息,以及其中存在哪些细微差别和偏差,十分重要。在本研究中,我们使用MS-COCO的重新标注版本Sama-COCO,借助形状分析流水线来发现潜在偏差。我们在两个数据集上分别训练并评估模型,以检验不同标注条件的影响。结果表明,标注风格至关重要,标注流水线应紧密结合目标任务加以考量。该数据集已在https://www.sama.com/sama-coco-dataset/ 公开发布。
英文摘要:
Benchmark datasets are used to profile and compare algorithms across a variety of tasks, ranging from image classification to segmentation, and also play a large role in image pretraining algorithms. Emphasis is placed on results with little regard to the actual content within the dataset. It is important to question what kind of information is being learned from these datasets and what are the nuances and biases within them. In the following work, Sama-COCO, a re-annotation of MS-COCO, is used to discover potential biases by leveraging a shape analysis pipeline. A model is trained and evaluated on both datasets to examine the impact of different annotation conditions. Results demonstrate that annotation styles are important and that annotation pipelines should closely consider the task of interest. The dataset is made publicly available at https://www.sama.com/sama-coco-dataset/ .
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。