USE-Evaluator：具有不确定、小或空参考标注的医学图像分割模型的性能指标
USE-Evaluator: Performance Metrics for Medical Image Segmentation Models with Uncertain, Small or Empty Reference Annotations
AI总结：
本文提出USE-Evaluator，研究不确定、小或空参考标注对医学图像分割指标的影响，发现常见指标在临床数据上失效，呼吁重新思考评估与优化程序。
摘要：
医学图像分割模型的性能指标用于衡量参考标注与预测分割之间的一致性。通常，重叠指标（如 Dice）被用作评估这些模型性能的指标，以使结果具有可比性。然而，与临床实践相比，公开数据集中病例的分布和分割任务的难度水平存在不匹配。常见指标无法衡量这种不匹配的影响，特别是对于包含低信号病变、困难的分割任务以及不确定、小或空参考标注的临床数据集。这一局限性可能导致机器学习从业者在设计和优化模型时进行无效的研究。评估临床价值的维度包括考虑参考标注的不确定性、不受参考标注体积大小的影响，以及对空参考标注分类的评估。我们在内部数据集上研究了不确定、小和空参考标注如何影响医学图像分割的指标价值，且不依赖于特定模型。我们检查了标准深度学习框架预测结果的指标行为，以识别具有临床价值的指标。我们将其与具有高信号病变以及确定、较大且无空参考标注的公开基准数据集（BraTS 2019）进行了比较。我们向机器学习从业者展示了不确定、小或空参考标注如何要求重新思考评估和优化程序。评估代码已发布以鼓励对该主题的进一步分析。https://github.com/SophieOstmeier/UncertainSmallEmpty.git
英文摘要：
Performance metrics for medical image segmentation models are used to measure the agreement between the reference annotation and the predicted segmentation. Usually, overlap metrics, such as the Dice, are used as a metric to evaluate the performance of these models in order for results to be comparable. However, there is a mismatch between the distributions of cases and difficulty level of segmentation tasks in public data sets compared to clinical practice. Common metrics fail to measure the impact of this mismatch, especially for clinical data sets that include low signal pathologies, a difficult segmentation task, and uncertain, small, or empty reference annotations. This limitation may result in ineffective research of machine learning practitioners in designing and optimizing models. Dimensions of evaluating clinical value include consideration of the uncertainty of reference annotations, independence from reference annotation volume size, and evaluation of classification of empty reference annotations. We study how uncertain, small, and empty reference annotations influence the value of metrics for medical image segmentation on an in-house data set regardless of the model. We examine metrics behavior on the predictions of a standard deep learning framework in order to identify metrics with clinical value. We compare to a public benchmark data set (BraTS 2019) with a high-signal pathology and certain, larger, and no empty reference annotations. We may show machine learning practitioners, how uncertain, small, or empty reference annotations require a rethinking of the evaluation and optimizing procedures. The evaluation code was released to encourage further analysis of this topic. https://github.com/SophieOstmeier/UncertainSmallEmpty.git
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
