Out-of-Distribution Detection and Data Drift Monitoring using Statistical Process Control
Office of Science and Engineering Laboratories（科学与工程实验室办公室）；Center for Devices and Radiological Health（设备与放射健康中心）；U.S. Food and Drug Administration（美国食品药品监督管理局）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Maryland（马里兰大学）；University of Maryland School of Medicine（马里兰大学医学院）
Ghada Zamzmi, Kesavan Venkatesh, Brandon Nelson, Smriti Prathapan, Paul H. Yi, Berkman Sahiner, Jana G. Delfino
Background: Machine learning (ML) methods often fail with data that deviates from their training distribution. This is a significant concern for ML-enabled devices in clinical settings, where data drift may cause unexpected performance that jeopardizes patient safety.
Method: We propose a ML-enabled Statistical Process Control (SPC) framework for out-of-distribution (OOD) detection and drift monitoring. SPC is advantageous as it visually and statistically highlights deviations from the expected distribution. To demonstrate the utility of the proposed framework for monitoring data drift in radiological images, we investigated different design choices, including methods for extracting feature representations, drift quantification, and SPC parameter selection.
Results: We demonstrate the effectiveness of our framework for two tasks: 1) differentiating axial vs. non-axial computed tomography (CT) images and 2) separating chest x-ray (CXR) from other modalities. For both tasks, we achieved high accuracy in detecting OOD inputs, with 0.913 in CT and 0.995 in CXR, and sensitivity of 0.980 in CT and 0.984 in CXR. Our framework was also adept at monitoring data streams and identifying the time a drift occurred. In a simulation with 100 daily CXR cases, we detected a drift in OOD input percentage from 0-1% to 3-5% within two days, maintaining a low false-positive rate. Through additional experimental results, we demonstrate the framework's data-agnostic nature and independence from the underlying model's structure.
Conclusion: We propose a framework for OOD detection and drift monitoring that is agnostic to data, modality, and model. The framework is customizable and can be adapted for specific applications.
Agent-Based Output Drift Detection for Breast Cancer Response Prediction in a Multisite Clinical Decision Support System
Quibim（Quibim公司）；Universitat de Barcelona（巴塞罗那大学）；Institució Catalana de Recerca i Estudis Avançats (ICREA)（加泰罗尼亚研究与高等教育学院（ICREA））；Computer Vision Center（计算机视觉中心）
Online Detection of Water Contamination Under Concept Drift
KIOS Research and Innovation Center of Excellence（KIOS研究与创新中心卓越中心）；Department of Electrical and Computer Engineering（电气与计算机工程系）；University of Cyprus（塞浦路斯大学）
AutoML for Multi-Class Anomaly Compensation of Sensor Drift
Institute for Information Processing (tnt), Leibniz University Hannover（汉诺威莱布尼茨大学信息处理研究所（tnt））；Department of Electrical and Computer Engineering, University of Southern California（南加州大学电气与计算机工程系）；Institute for Artificial Intelligence, Leibniz University Hannover（汉诺威莱布尼茨大学人工智能研究所）
Trustworthy Self-Composable Big-Data-as-a-Service: An LLM-Orchestrated Multi-Agent Framework for Automated Data Engineering, AutoML, MLOps Deployment, and Drift-Aware Lifecycle Optimization
School of Information, Computer, and Communication Technology（信息、计算机与通信技术学院）；Sirindhorn International Institute of Technology, Thammasat University（素金国际技术研究所，泰国 Thammasat 大学）
