使用未标记数据的在线性能估计：Hui-Walter范式的贝叶斯应用
Online Performance Estimation with Unlabeled Data: A Bayesian Application of the Hui-Walter Paradigm
AI总结：
本文将Hui-Walter范式引入机器学习，通过Gibbs采样和交叉制表在无标签和动态数据环境下估计假阳性率等性能指标，实现了无需真实标签的在线模型评估。
摘要：
在机器学习和统计建模的工业实践中，从业者通常假设存在可访问的、静态的带标签数据用于评估和训练。然而，这一假设往往偏离现实，数据可能是私有的、加密的、难以测量的或未标记的。在本文中，我们通过将传统上应用于流行病学和医学的Hui-Walter范式调整到机器学习领域来弥合这一差距。这种方法使我们能够在没有真实标签可用的情况下，估计关键性能指标，如假阳性率、假阴性率和先验概率。我们进一步扩展了这一范式以处理在线数据，为动态数据环境开辟了新的可能性。我们的方法包括将数据划分为潜在类别以模拟多个数据群体（如果没有自然群体的话），并独立训练模型以复制多个测试。通过对多个分类器和多个群体的二元结果进行交叉制表，我们能够通过Gibbs采样估计未知参数，从而无需真实标签或带标签数据。本文展示了我们的方法论通过在动态和不确定的数据条件下允许准确的模型评估，从而改变机器学习实践的潜力。
英文摘要：
In the industrial practice of machine learning and statistical modeling, practitioners often work under the assumption of accessible, static, labeled data for evaluation and training. However, this assumption often deviates from reality, where data may be private, encrypted, difficult-to-measure, or unlabeled. In this paper, we bridge this gap by adapting the Hui-Walter paradigm, a method traditionally applied in epidemiology and medicine, to the field of machine learning. This approach enables us to estimate key performance metrics such as false positive rate, false negative rate, and priors in scenarios where no ground truth is available. We further extend this paradigm for handling online data, opening up new possibilities for dynamic data environments. Our methodology involves partitioning data into latent classes to simulate multiple data populations (if natural populations are unavailable) and independently training models to replicate multiple tests. By cross-tabulating binary outcomes across multiple categorizers and multiple populations, we are able to estimate unknown parameters through Gibbs sampling, eliminating the need for ground-truth or labeled data. This paper showcases the potential of our methodology to transform machine learning practices by allowing for accurate model assessment under dynamic and uncertain data conditions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
