超越排行榜:解决研究问题的洞察力和部署挑战
Beyond the Leaderboard: Insight and Deployment Challenges to Address Research Problems
摘要:
在医学图像分析领域,从2007年开始在国际会议上组织相关研讨会的挑战,并已发展到包括150多个挑战。其中一些挑战在实地产生了重大影响。然而,尽管设计良好的挑战有可能将该领域团结起来,并专注于为重要问题创造解决方案,但设计不良和记录不良的挑战同样会阻碍一个领域,并导致追求无理论或临床意义的指标评分的渐进式改进。这一点得到了国际MICCAI会议对挑战的批判性评估的支持。在该评估中,主要观察结果是,基本挑战数据的微小变化可能会彻底改变排行榜上的排名顺序。与此相关的是排行榜攀登的做法,其特点是参与者专注于逐步改善指标结果,而不是推进科学或解决挑战的驱动问题。在这篇摘要中,我们超越了挑战的排行榜,而是着眼于从挑战中得出的关于它所解决的研究问题的结论。研究设计在其他研究领域得到了很好的描述,当将挑战视为解决研究问题的算法性能研究时,可以将其转化为挑战设计。基于科学研究设计的两种主要类型,我们提出了两种主要的挑战类型,我们认为这也将有利于其他研究领域:1)基于定性研究设计的洞察力挑战和2)基于定量研究设计的部署挑战。此外,我们还简要地谈到了统计意义与实际意义、可推广性和数据饱和度的相关考虑。
英文摘要:
In the medical image analysis field, organizing challenges with associated workshops at international conferences began in 2007 and has grown to include over 150 challenges. Several of these challenges have had a major impact in the field. However, whereas well-designed challenges have the potential to unite and focus the field on creating solutions to important problems, poorly designed and documented challenges can equally impede a field and lead to pursuing incremental improvements in metric scores with no theoretic or clinical significance. This is supported by a critical assessment of challenges at the international MICCAI conference. In this assessment the main observation was that small changes to the underlying challenge data can drastically change the ranking order on the leaderboard. Related to this is the practice of leaderboard climbing, which is characterized by participants focusing on incrementally improving metric results rather than advancing science or solving the driving problem of a challenge. In this abstract we look beyond the leaderboard of a challenge and instead look at the conclusions that can be drawn from a challenge with respect to the research problem that it is addressing. Research study design is well described in other research areas and can be translated to challenge design when viewing challenges as research studies on algorithm performance that address a research problem. Based on the two main types of scientific research study design, we propose two main challenge types, which we think would benefit other research areas as well: 1) an insight challenge that is based on a qualitative study design and 2) a deployment challenge that is based on a quantitative study design. In addition we briefly touch upon related considerations with respect to statistical significance versus practical significance, generalizability and data saturation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。