文本数据标注的自适应规则发现
Adaptive Rule Discovery for Labeling Text Data
摘要：
创建和收集标记数据是机器学习流水线中的主要瓶颈之一，并且自动特征生成技术（诸如深度学习）的出现（其通常需要大量训练数据）进一步加剧了该问题。虽然弱监督技术已经绕过了这一瓶颈，但现有的框架要么要求用户编写一组多样化的高质量规则来标记数据（例如，Snorkel），或者需要数据的带标签子集来自动挖掘规则（例如，Snuba）。手动编写规则的过程可能是繁琐和耗时的。与此同时，创建数据的标记子集可能成本高昂，甚至在不平衡的环境中不可行。这是因为在不平衡环境中的随机样本通常只包含少数正面实例。为了解决这些缺点，我们提出了达尔文，一个交互式系统，旨在减轻弱监督设置中的文本数据标记规则的编写任务。给定一个初始的标注规则，Darwin会自动生成一组候选规则用于当前的标注任务，并利用注释者的反馈来调整候选规则。我们描述了达尔文是如何可扩展和多功能的。它可以在大型文本语料库（即，超过100万个句子）并且支持广泛的标记功能（即，可以使用上下文无关文法指定的任何函数）。最后，我们通过一系列实验证明了五个真实世界的数据集，达尔文使注释器能够有效地生成弱监督标签，并且成本很低。事实上，我们的实验表明，达尔文发现的规则平均识别40%以上的积极实例相比，Snuba，即使它提供了1000个标记的实例。
英文摘要：
Creating and collecting labeled data is one of the major bottlenecks in machine learning pipelines and the emergence of automated feature generation techniques such as deep learning, which typically requires a lot of training data, has further exacerbated the problem. While weak-supervision techniques have circumvented this bottleneck, existing frameworks either require users to write a set of diverse, high-quality rules to label data (e.g., Snorkel), or require a labeled subset of the data to automatically mine rules (e.g., Snuba). The process of manually writing rules can be tedious and time consuming. At the same time, creating a labeled subset of the data can be costly and even infeasible in imbalanced settings. This is due to the fact that a random sample in imbalanced settings often contains only a few positive instances. To address these shortcomings, we present Darwin, an interactive system designed to alleviate the task of writing rules for labeling text data in weakly-supervised settings. Given an initial labeling rule, Darwin automatically generates a set of candidate rules for the labeling task at hand, and utilizes the annotator's feedback to adapt the candidate rules. We describe how Darwin is scalable and versatile. It can operate over large text corpora (i.e., more than 1 million sentences) and supports a wide range of labeling functions (i.e., any function that can be specified using a context free grammar). Finally, we demonstrate with a suite of experiments over five real-world datasets that Darwin enables annotators to generate weakly-supervised labels efficiently and with a small cost. In fact, our experiments show that rules discovered by Darwin on average identify 40% more positive instances compared to Snuba even when it is provided with 1000 labeled instances.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
