通过演示的数据编程:交互式学习标记函数的框架
Data Programming by Demonstration: A Framework for Interactively Learning Labeling Functions
摘要:
数据编程是一种程序化的弱监督方法,可以有效地管理大规模标记的训练数据。然而,编写数据程序(标记函数)需要编程素养和领域专业知识。许多主题专家既不精通编程,也没有时间有效地编写数据程序。此外,无论一个人在编码或机器学习方面的专业知识如何,通过枚举规则和阈值将领域专业知识转换为标记函数不仅耗时而且固有地困难。在这里,我们提出了一个新的框架,数据编程演示(DPBD),生成标签规则,使用用户的交互式演示。DPBD旨在减轻用户编写标记函数的负担,使他们能够专注于更高级别的语义,例如识别标记任务的相关信号。我们操作我们的框架与统治者,一个交互式系统,综合标签规则的文档分类,使用跨级注释的用户文档的例子。我们通过一项用户研究将Ruler与传统的数据编程进行了比较,该研究由10位数据科学家为情感和垃圾邮件分类任务创建标签函数。我们发现,统治者更容易使用和学习,并提供更高的整体满意度,同时提供判别模型的性能相比,通过传统的数据编程实现。
英文摘要:
Data programming is a programmatic weak supervision approach to efficiently curate large-scale labeled training data. Writing data programs (labeling functions) requires, however, both programming literacy and domain expertise. Many subject matter experts have neither programming proficiency nor time to effectively write data programs. Furthermore, regardless of one's expertise in coding or machine learning, transferring domain expertise into labeling functions by enumerating rules and thresholds is not only time consuming but also inherently difficult. Here we propose a new framework, data programming by demonstration (DPBD), to generate labeling rules using interactive demonstrations of users. DPBD aims to relieve the burden of writing labeling functions from users, enabling them to focus on higher-level semantics such as identifying relevant signals for labeling tasks. We operationalize our framework with Ruler, an interactive system that synthesizes labeling rules for document classification by using span-level annotations of users on document examples. We compare Ruler with conventional data programming through a user study conducted with 10 data scientists creating labeling functions for sentiment and spam classification tasks. We find that Ruler is easier to use and learn and offers higher overall satisfaction, while providing discriminative model performances comparable to ones achieved by conventional data programming.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。