首页/论文/1809.00537

arXiv 1809.005372018-09-03cs.CL

Crowdsourcing Semantic Label Propagation in Relation Classification

Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；IBM CAS Benelux（IBM比荷卢系统中心）；Google Research（谷歌研究院）

Anca Dumitrache, Lora Aroyo, Chris Welty

备注：: In publication at the First Workshop on Fact Extraction and Verification (FeVer) at EMNLP 2018
DOI：: 10.18653/v1/W18-5503

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

Distant supervision is a popular method for performing relation extraction from text that is known to produce noisy labels. Most progress in relation extraction and classification has been made with crowdsourced corrections to distant-supervised labels, and there is evidence that indicates still more would be better. In this paper, we explore the problem of propagating human annotation signals gathered for open-domain relation classification through the CrowdTruth methodology for crowdsourcing, that captures ambiguity in annotations by measuring inter-annotator disagreement. Our approach propagates annotations to sentences that are similar in a low dimensional embedding space, expanding the number of labels by two orders of magnitude. Our experiments show significant improvement in a sentence-level multi-class relation classifier.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

基于跨度序列标注的成本高效众包：工作者选择与数据增强

Cost-efficient Crowdsourcing for Span-based Sequence Labeling: Worker Selection and Data Augmentation

School of Computer and Information Technology, Beijing Jiaotong University（北京交通大学计算机与信息技术学院）；Institute for Interdisciplinary Information Sciences, Tsinghua University（清华大学交叉信息研究院）；School of Information Science, Beijing Language and Culture University（北京语言大学信息科学学院）；Department of Computer Science, The University of California, Davis（加州大学戴维斯分校计算机科学系）

本文提出一种基于CMAB的众包工作者选择算法及SES数据增强方法，有效应对序列标注中的标签依赖问题，在提升F1分数的同时大幅降低成本，并可无缝集成至RLHF系统。

2024-07-27cs.AIcs.CL

Crowdsourcing Ground Truth for Medical Relation Extraction

Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；Google Research（谷歌研究院）；CAS IBM Netherlands（荷兰IBM认知计算系统研究中心）

Cognitive computing systems require human labeled data for evaluation, and often for training. The standard practice used in gathering this data minimizes disagreement between annotators, and we have found this results in data that fails to account for the ambiguity inherent in language. We have proposed the CrowdTruth method for collecting ground truth through crowdsourcing, that reconsiders the role of people in machine learning based on the observation that disagreement between annotators provides a useful signal for phenomena such as ambiguity in the text. We report on using this method to…

2017-10-03cs.CLcs.HC

A Bayesian Approach for Sequence Tagging with Crowds

Technische Universität Darmstadt（达姆施塔特工业大学）

Current methods for sequence tagging, a core task in NLP, are data hungry, which motivates the use of crowdsourcing as a cheap way to obtain labelled data. However, annotators are often unreliable and current aggregation methods cannot capture common types of span annotation errors. To address this, we propose a Bayesian method for aggregating sequence tags that reduces errors by modelling sequential dependencies between the annotations as well as the ground-truth labels. By taking a Bayesian approach, we account for uncertainty in the model due to both annotator errors and the lack of data fo…

2019-09-06cs.CLcs.LG

通过标签聚合对众包与大语言模型标注质量的比较研究

A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation

University of Yamanashi（山梨大学）

本文构建众包数据集基准，比较个体众包与 LLM 标签及聚合标签质量，并提出 Crowd-LLM 混合聚合方法，证明加入优质 LLM 标签可提升众包聚合标签质量。

2024-01-18cs.CLcs.HC

扩大公共卫生文本标注规模：零样本学习与众包的效率提升和标注准确性对比

Scaling Public Health Text Annotation: Zero-Shot Learning vs. Crowdsourcing for Improved Efficiency and Labeling Accuracy

University of Toronto（多伦多大学）；Dalla Lana School of Public Health（达拉·拉纳公共卫生学院）；Institute of Health Policy, Management, and Evaluation (IHPME)（卫生政策、管理与评价研究所）；Faculty of Information（信息学院）；Schwartz Reisman Institute（施瓦茨·赖斯曼研究所）

本研究针对公共卫生社交媒体数据标注耗时昂贵的问题，对比零样本LLM标注与传统众包标注的表现，发现LLM在简单分类任务中可媲美人类且大幅提速，但复杂任务准确性下降，为公共卫生研究标注提供高效方案。

2025-02-10cs.CL

Crowdsourcing Universal Part-Of-Speech Tags for Code-Switching

Columbia University（哥伦比亚大学）

Code-switching is the phenomenon by which bilingual speakers switch between multiple languages during communication. The importance of developing language technologies for codeswitching data is immense, given the large populations that routinely code-switch. High-quality linguistic annotations are extremely valuable for any NLP task, and performance is often limited by the amount of high-quality labeled data. However, little such data exists for code-switching. In this paper, we describe crowd-sourcing universal part-of-speech tags for the Miami Bangor Corpus of Spanish-English code-switched s…

2017-03-24cs.CL

更多相关论文收起更多论文

Towards A Reliable Ground-Truth For Biased Language Detection

University of Konstanz（康斯坦茨大学）；University of Wuppertal（伍珀塔尔大学）

Reference texts such as encyclopedias and news articles can manifest biased language when objective reporting is substituted by subjective writing. Existing methods to detect bias mostly rely on annotated data to train machine learning models. However, low annotator agreement and comparability is a substantial drawback in available media bias corpora. To evaluate data collection options, we collect and compare labels obtained from two popular crowdsourcing platforms. Our results demonstrate the existing crowdsourcing approaches' lack of data quality, underlining the need for a trained expert f…

2021-12-17cs.CL

Crowdsourcing Question-Answer Meaning Representations

Bar-Ilan University（巴伊兰大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）

We introduce Question-Answer Meaning Representations (QAMRs), which represent the predicate-argument structure of a sentence as a set of question-answer pairs. We also develop a crowdsourcing scheme to show that QAMRs can be labeled with very little training, and gather a dataset with over 5,000 sentences and 100,000 questions. A detailed qualitative analysis demonstrates that the crowd-generated question-answer pairs cover the vast majority of predicate-argument relationships in existing datasets (including PropBank, NomBank, QA-SRL, and AMR) along with many previously under-resourced ones, i…

2017-11-16cs.CL

Visualizing NLP annotations for Crowdsourcing

University of Washington（华盛顿大学）

Visualizing NLP annotation is useful for the collection of training data for the statistical NLP approaches. Existing toolkits either provide limited visual aid, or introduce comprehensive operators to realize sophisticated linguistic rules. Workers must be well trained to use them. Their audience thus can hardly be scaled to large amounts of non-expert crowdsourced workers. In this paper, we present CROWDANNO, a visualization toolkit to allow crowd-sourced workers to annotate two general categories of NLP problems: clustering and parsing. Workers can finish the tasks with simplified operators…

2015-08-25cs.CL

Controlled Crowdsourcing for High-Quality QA-SRL Annotation

Bar-Ilan University（巴伊兰大学）；Intel AI Lab（英特尔人工智能实验室）；Paul G. Allen School of Computer Science & Engineering（保罗·G·艾伦计算机科学与工程学院）；Allen Institute for AI（艾伦人工智能研究所）；Facebook AI Research（脸书人工智能研究院）；University of Washington（华盛顿大学）

Question-answer driven Semantic Role Labeling (QA-SRL) was proposed as an attractive open and natural flavour of SRL, potentially attainable from laymen. Recently, a large-scale crowdsourced QA-SRL corpus and a trained parser were released. Trying to replicate the QA-SRL annotation for new texts, we found that the resulting annotations were lacking in quality, particularly in coverage, making them insufficient for further research and evaluation. In this paper, we present an improved crowdsourcing protocol for complex semantic annotation, involving worker selection and training, and a data con…

2020-05-13cs.CL

论话语关系标注的众包任务设计

On Crowdsourcing Task Design for Discourse Relation Annotation

Saarland University（萨尔大学）

本研究比较了自由选择与强制选择两种众包标注方法，发现自由选择虽灵活但标注多样性低，并揭示了任务设计与标注者能力间的相互作用。

2024-12-16cs.CL

协同LLM与全局标签传播的多模态假新闻检测

Synergizing LLMs with Global Label Propagation for Multimodal Fake News Detection

Inner Mongolia University（内蒙古大学）；National University of Singapore（新加坡国立大学）

针对LLM伪标签性能差的问题，提出GLPN-LLM模型，通过全局标签传播与掩码机制整合LLM能力，在多模态假新闻检测中超越现有基线。

2025-05-31cs.CL

Toward Effective Automated Content Analysis via Crowdsourcing

Beijing Institute of Technology（北京理工大学）；North Carolina State University（北卡罗来纳州立大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）

Many computer scientists use the aggregated answers of online workers to represent ground truth. Prior work has shown that aggregation methods such as majority voting are effective for measuring relatively objective features. For subjective features such as semantic connotation, online workers, known for optimizing their hourly earnings, tend to deteriorate in the quality of their responses as they work longer. In this paper, we aim to address this issue by proposing a quality-aware semantic data annotation system. We observe that with timely feedback on workers' performance quantified by qual…

2021-04-04cs.CLcs.IRcs.LG

Supervised Topical Key Phrase Extraction of News Stories using Crowdsourcing, Light Filtering and Co-reference Normalization

CMU（卡内基梅隆大学）；INESC-IST

Fast and effective automated indexing is critical for search and personalized services. Key phrases that consist of one or more words and represent the main concepts of the document are often used for the purpose of indexing. In this paper, we investigate the use of additional semantic features and pre-processing steps to improve automatic key phrase extraction. These features include the use of signal words and freebase categories. Some of these features lead to significant improvements in the accuracy of the results. We also experimented with 2 forms of document pre-processing that we call l…

2013-06-20cs.CLcs.IR