预测驱动的已部署模型风险监控:检测有害分布漂移
Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shifts
AI总结:
提出预测驱动风险监控(PPRM),一种基于预测驱动推断的半监督方法,通过结合合成标签与少量真实标签构建运行风险的随时有效下界,实现对有害漂移的检测,并在图像分类、大语言模型和电信监控任务中验证有效性。
摘要:
我们研究了在动态环境中模型性能监控的问题,其中标记数据有限。为此,我们提出了预测驱动风险监控(PPRM),一种基于预测驱动推断(PPI)的半监督风险监控方法。PPRM通过结合合成标签与少量真实标签,构建运行风险的随时有效下界。通过基于阈值的比较与名义风险的上界,检测有害漂移,满足无假设的有限样本I型误差保证。我们通过在图像分类、大语言模型(LLM)和电信监控任务上的大量实验,证明了PPRM的有效性。
英文摘要:
We study the problem of monitoring model performance in dynamic environments where labeled data are limited. To this end, we propose prediction-powered risk monitoring (PPRM), a semi-supervised risk-monitoring approach based on prediction-powered inference (PPI). PPRM constructs anytime-valid lower bounds on the running risk by combining synthetic labels with a small set of true labels. Harmful shifts are detected via a threshold-based comparison with an upper bound on the nominal risk, satisfying assumption-free finite-sample guarantees on the type-I error. We demonstrate the effectiveness of PPRM through extensive experiments on image classification, large language model (LLM), and telecommunications monitoring tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。