arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.11010cs.LGcs.GT
收藏

WeShap:基于 Shapley 值的弱监督源评估

WeShap: Weak Supervision Source Evaluation with Shapley Values

University of Toronto(多伦多大学)

Naiqing Guan, Nick Koudas

AI总结:

本文提出 WeShap,利用 Shapley 值和动态规划评估弱监督源贡献,可筛选标注函数、纠错并改进 PWS,使下游准确率平均提升 5.0 点。

摘要:

高效数据标注是训练当代机器学习模型的一个重要瓶颈。程序化弱监督(PWS)流程提供了一种解决方案,它利用多个弱监督源自动标注数据,从而加快标注过程。鉴于这些弱监督源对 PWS 准确性的贡献各不相同,必须采用稳健且高效的指标对其进行评估。这不仅对于理解 PWS 流程的行为和性能至关重要,也有助于采取纠正措施。在我们的研究中,我们提出 WeShap 值作为一种评估指标;该指标借助 Shapley 值的理论基础,量化弱监督源在代理 PWS 流程中的平均贡献。我们展示了如何使用动态规划高效计算 WeShap 值,其计算复杂度相对于弱监督源数量达到二次级别。实验证明 WeShap 值可广泛用于多种应用,包括识别有益或有害的标注函数、改进 PWS 流程以及纠正错误标注数据。此外,WeShap 值有助于理解 PWS 流程的行为,并审查错误标注数据的具体实例。尽管 WeShap 值最初源自一个特定的代理 PWS 流程,但我们通过实验证明它可推广到其他 PWS 流程配置。研究结果表明,通过修订 PWS 流程,与以往最先进方法相比,下游模型准确率平均显著提升 5.0 个点,凸显了 WeShap 值在提升机器学习模型训练数据质量方面的有效性。

英文摘要:

Efficient data annotation stands as a significant bottleneck in training contemporary machine learning models. The Programmatic Weak Supervision (PWS) pipeline presents a solution by utilizing multiple weak supervision sources to automatically label data, thereby expediting the annotation process. Given the varied contributions of these weak supervision sources to the accuracy of PWS, it is imperative to employ a robust and efficient metric for their evaluation. This is crucial not only for understanding the behavior and performance of the PWS pipeline but also for facilitating corrective measures. In our study, we introduce WeShap values as an evaluation metric, which quantifies the average contribution of weak supervision sources within a proxy PWS pipeline, leveraging the theoretical underpinnings of Shapley values. We demonstrate efficient computation of WeShap values using dynamic programming, achieving quadratic computational complexity relative to the number of weak supervision sources. Our experiments demonstrate the versatility of WeShap values across various applications, including the identification of beneficial or detrimental labeling functions, refinement of the PWS pipeline, and rectification of mislabeled data. Furthermore, WeShap values aid in comprehending the behavior of the PWS pipeline and scrutinizing specific instances of mislabeled data. Although initially derived from a specific proxy PWS pipeline, we empirically demonstrate the generalizability of WeShap values to other PWS pipeline configurations. Our findings indicate a noteworthy average improvement of 5.0 points in downstream model accuracy through the revision of the PWS pipeline compared to previous state-of-the-art methods, underscoring the efficacy of WeShap values in enhancing data quality for training machine learning models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑