Many machine learning systems today are trained on large amounts of human-annotated data. Data annotation tasks that require a high level of competency make data acquisition expensive, while the resulting labels are often subjective, inconsistent, and may contain a variety of human biases. To improve the data quality, practitioners often need to collect multiple annotations per example and aggregate them before training models. Such a multi-stage approach results in redundant annotations and may often produce imperfect "ground truth" that may limit the potential of training accurate machine learning models. We propose a new end-to-end framework that enables us to: (i) merge the aggregation step with model training, thus allowing deep learning systems to learn to predict ground truth estimates directly from the available data, and (ii) model difficulties of examples and learn representations of the annotators that allow us to estimate and take into account their competencies. Our approach is general and has many applications, including training more accurate models on crowdsourced data, ensemble learning, as well as classifier accuracy estimation from unlabeled data. We conduct an extensive experimental evaluation of our method on 5 crowdsourcing datasets of varied difficulty and show accuracy gains of up to 25% over the current state-of-the-art approaches for aggregating annotations, as well as significant reductions in the required annotation redundancy.
Prototype-Anchored Learning for Learning with Imperfect Annotations
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Tsinghua University（清华大学）
Reinforcement Learning From Imperfect Corrective Actions And Proxy Rewards
The University of Michigan-Shanghai Jiao Tong University Joint Institute (UM-SJTU JI)（密歇根大学-上海交通大学联合学院）；Duke Kunshan University（昆山杜克大学）；NetEase Fuxi AI Lab（网易伏羲人工智能实验室）
HyPAC: Cost-Efficient LLMs-Human Hybrid Annotation with PAC Error Guarantees
Department of Statistics and Data Science, Southern University of Science and Technology, China（统计与数据科学系，南方科技大学）；Department of Statistics, University of California at Riverside, USA（统计系，加州大学河滨分校）；College of Computing and Data Science, Nanyang Technological University, Singapore（计算与数据科学学院，南洋理工大学）；School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China（人工智能学院，香港中文大学（深圳））
Deep Active Re-Labeling: Toward Noise-Resilient Annotation Efficiency
Active Machine Learning Lab（主动学习实验室）；Learning · Language · Vision (LLaVi) Lab（学习·语言·视觉（LLaVi）实验室）；Department of Computer Science and Engineering（计算机科学与工程系）
Disentangling Imperfect: A Wavelet-Infused Multilevel Heterogeneous Network for Human Activity Recognition in Flawed Wearable Sensor Data
School of Computer Science and Engineering, Tianjin University of Technology（天津理工大学计算机科学与工程学院）；Technical University of Denmark（丹麦技术大学）；Aalborg University（奥尔堡大学）
An Evaluation of Hybrid Annotation Workflows on High-Ambiguity Spatiotemporal Video Footage
IPTC（信息处理与电信中心）；Telecommunications Center, Escuela Técnica Superior de Ingenieros de Telecomunicación, Av. Complutense 30, 28040 Madrid, Spain（电信中心，电信工程师高级学院，Complutense大道30号，28040马德里，西班牙）
