评估预测系统并减少其警报负担的新技术
Novel Techniques to Assess Predictive Systems and Reduce Their Alarm Burden
摘要:
机器预测算法(例如,二进制分类器)通常是基于使用诸如灵敏度和预测值之类的经典度量所声称的性能来采用的。然而,分类器的性能在很大程度上取决于分类器操作的上下文(工作流)。经典指标不反映预测因子的实际效用,除非满足某些隐含的假设,而这些假设在许多常见的临床场景中无法满足。这往往会导致次优的执行和失望时,预期的成果没有实现。经典指标的一个常见故障模式出现在可以对同一事件进行多个预测时,特别是当冗余的真阳性预测产生很少的附加值时。这描述了许多临床警报系统。我们解释了为什么经典的指标不能正确地表示预测性能在这样的背景下,并介绍了一种改进的性能评估技术,使用效用函数来评分预测的基础上,他们的效用在一个特定的工作流上下文中。由此产生的效用度量(u-度量)明确占时间关系的预测效用的影响。与传统测量相比,u度量更准确地反映了在现场临床环境中操作的预测器的真实世界成本和收益。这种改进可能是显著的。我们还描述了一个正式的方法打盹,缓解策略,其中一些预测被抑制,以提高预测性能,减少误报,同时保留事件捕获。Snoozing对于生成中断警报的预测器特别有用。U-指标正确地测量和预测了休眠的性能优势,而传统的指标却不能。
英文摘要:
Machine prediction algorithms (e.g., binary classifiers) often are adopted on the basis of claimed performance using classic metrics such as sensitivity and predictive value. However, classifier performance depends heavily upon the context (workflow) in which the classifier operates. Classic metrics do not reflect the realized utility of a predictor unless certain implicit assumptions are met, and these assumptions cannot be met in many common clinical scenarios. This often results in suboptimal implementations and in disappointment when expected outcomes are not achieved. One common failure mode for classic metrics arises when multiple predictions can be made for the same event, particularly when redundant true positive predictions produce little additional value. This describes many clinical alerting systems. We explain why classic metrics cannot correctly represent predictor performance in such contexts, and introduce an improved performance assessment technique using utility functions to score predictions based on their utility in a specific workflow context. The resulting utility metrics (u-metrics) explicitly account for the effects of temporal relationships on prediction utility. Compared to traditional measures, u-metrics more accurately reflect the real world costs and benefits of a predictor operating in a live clinical context. The improvement can be significant. We also describe a formal approach to snoozing, a mitigation strategy in which some predictions are suppressed to improve predictor performance by reducing false positives while retaining event capture. Snoozing is especially useful for predictors that generate interruptive alarms. U-metrics correctly measure and predict the performance benefits of snoozing, whereas traditional metrics do not.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。