基于设计的带有噪声人类标签的监督学习
Design-Based Supervised Learning with Noisy Human Labels
- RTI International（RTI国际）
- U.S. Bureau of Labor Statistics（美国劳动统计局）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
研究针对自动分类器标记非结构化数据时人类审核标签有噪声的问题，提出PA-DSL方法，利用裁决案例纠正标签并校正分析偏差，在合成和维基百科解毒半合成实验中取得降低均方根误差10%-17%的效果。
AI中文摘要：
研究人员越来越多地使用自动分类器为统计分析标记非结构化数据。现有纠正方法可利用概率抽样审核集纠正自动标签中的错误，但通常将审核标签视为正确的。实际上，人工审核标签常存在噪声，且只有部分审核项目由专家或裁决者复查。我们提出了部分裁决的基于设计的监督学习（PA-DSL）方法。它利用裁决案例纠正有噪声的人类标签，然后用纠正后的审核信息对基于全套自动标签的分析进行偏差校正。当审核和裁决概率已知时，该估计器对广泛的下游分析有效。在合成和维基百科解毒半合成实验中，当有噪声的人类标签包含可恢复信号时，PA-DSL保持名义覆盖率，相对于仅使用裁决标签，均方根误差降低了10%-17%。
英文摘要：
Researchers increasingly use automated classifiers to label unstructured data for statistical analysis. Existing rectification methods can correct errors in these automated labels using a probability-sampled audit set, but they usually treat the audit labels as correct. In practice, human audit labels are often noisy, and only some audited items are reviewed by an expert or adjudicator. We propose Partially Adjudicated Design-Based Supervised Learning (PA-DSL), a method for this setting. It uses adjudicated cases to correct noisy human labels and then uses the corrected audit information to debias analyses based on the full set of automated labels. The estimator is valid for a broad class of downstream analyses when the audit and adjudication probabilities are known. In synthetic and Wikipedia Detox semi-synthetic experiments, PA-DSL maintains nominal coverage and reduces RMSE by 10-17% relative to using only adjudicated labels when noisy human labels contain recoverable signal.
