Well-annotated datasets, as shown in recent top studies, are becoming more important for researchers than ever before in supervised machine learning (ML). However, the dataset annotation process and its related human labor costs remain overlooked. In this work, we analyze the relationship between the annotation granularity and ML performance in sequence labeling, using clinical records from nursing shift-change handover. We first study a model derived from textual language features alone, without additional information based on nursing knowledge. We find that this sequence tagger performs well in most categories under this granularity. Then, we further include the additional manual annotations by a nurse, and find the sequence tagging performance remaining nearly the same. Finally, we give a guideline and reference to the community arguing it is not necessary and even not recommended to annotate in detailed granularity because of a low Return on Investment. Therefore we recommend emphasizing other features, like textual knowledge, for researchers and practitioners as a cost-effective source for increasing the sequence labeling performance.
Exploring Optimal Granularity for Extractive Summarization of Unstructured Health Records: Analysis of the Largest Multi-Institutional Archive of Health Records in Japan
Tokyo Metropolitan University(东京都立大学);RIKEN(理化学研究所);National Hospital Organization(国立医院机构);Kitami Institute of Technology(北见工业大学)
MedSNIP: Building and Benchmarking Snippet-Level Granularity for Medical Fact Verification
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Yale University(耶鲁大学);Korea University(高丽大学);Beijing Normal University(北京师范大学);INSAIT, Sofia University, “St. Kliment Ohridski”(索非亚大学INSAIT研究所)
A Multi-granularity Concept Sparse Activation and Hierarchical Knowledge Graph Fusion Framework for Rare Disease Diagnosis
School of Software, Yunnan University(云南大学软件学院);Yunnan Key Laboratory of Software Engineering(云南软件工程重点实验室);Yunnan Provincial Hospital of Traditional Chinese Medicine(云南中医药省人民医院)
Cost-Quality Adaptive Active Learning for Chinese Clinical Named Entity Recognition
School of Information Science and Engineering, East China University of Science and Technology(华东理工大学信息科学与工程学院);Key Laboratory of Advanced Control and Optimization for Chemical Processes, Ministry of Education(化工过程先进控制和优化技术教育部重点实验室)
临床命名实体识别(Clinical Named Entity Recognition,CNER)的目的是自动识别电子健康档案(Electronic Health Records,EHR)中的临床术语,这是临床研究的基础和关键步骤。为CNER训练高性能模型,通常需要大量具有高质量标签的EHR。然而,给电子病历,尤其是中文版的电子病历贴上标签既费时又费钱。对此的一个有效解决方案是主动学习,其中模型要求标注器注释模型不确定的数据。传统的主动学习假设一个单一的标签,总是回答无声的答案查询标签。然而,在实际设置中,多个标注器以不同的成本提供不同质量的标注,并且总体标注质量较低的标注器仍然可以为某些特定实例分配正确的标签。在本文中,我们提出了一个成本-质量自适应主动学习(CQAAL)的方法CNER在中国电子病历,它保持了注释质量之间的平衡,标签成本,和所选实例的信息量。具体来说,CQAAL选择具有成本效益的实例-标签对,以自适应的方式以较低的成本实现更好的注释质量。CCKS-2017任务2基准数据集上的计算结果证明了所提出的CQAAL的优越性和有效性。
OPTIC: Optimizing Patient-Provider Triaging & Improving Communications in Clinical Operations using GPT-4 Data Labeling and Model Distillation
Microsoft Health AI(微软健康人工智能);Johns Hopkins University(约翰斯·霍普金斯大学);Health Information Technology(健康信息技术);Johns Hopkins Medicine(约翰斯·霍普金斯医学);University of Washington(华盛顿大学);Microsoft(微软)
PromptRad: Knowledge-Enhanced Multi-Label Prompt-Tuning for Low-Resource Radiology Report Labeling
Department of Artificial Intelligence and AI Research Center, Chang Gung University(人工智能系及AI研究中心,长庚大学);Department of Radiology, Sijhih Cathay General Hospital(放射科,西吉医院);Department of Medical Imaging and Intervention, Chang Gung Memorial Hospital(医学影像与介入科,长庚纪念医院);Department of Trauma and Emergency Surgery, Chang Gung Memorial Hospital(创伤与急诊外科,长庚纪念医院);Department of Computer Science, National Tsing Hua University(计算机科学系,国立清华大学)
Can Modern NLP Systems Reliably Annotate Chest Radiography Exams? A Pre-Purchase Evaluation and Comparative Study of Solutions from AWS, Google, Azure, John Snow Labs, and Open-Source Models on an Independent Pediatric Dataset
Automated SNOMED CT Concept Annotation in Clinical Text Using Bi-GRU Neural Networks
Informatics and Analytics, University of North Carolina Greensboro(信息学与分析,北卡罗来纳大学格林斯堡分校);Department of Computer Science, University of Nebraska Omaha(计算机科学系,内布拉斯加大学奥马哈分校)
Automated Labeling of German Chest X-Ray Radiology Reports using Deep Learning
Munich Institute of Biomedical Engineering, Technical University of Munich(慕尼黑工业大学慕尼黑生物医学工程研究所);School of Computation, Information and Technology, Technical University of Munich(慕尼黑工业大学计算、信息与技术学院);University Hospital, LMU Munich(慕尼黑大学附属医院);Institute of Neuroradiology, University Hospital, LMU Munich(慕尼黑大学附属医院神经放射学研究所)
What Do Patients Say About Their Disease Symptoms? Deep Multilabel Text Classification With Human-in-the-Loop Curation for Automatic Labeling of Patient Self Reports of Problems
RadAnnotate: Large Language Models for Efficient and Reliable Radiology Report Annotation
Department of Computer Science, University of California, Davis, CA, USA(加州大学戴维斯分校计算机科学系);Department of Radiology, University of California, Davis, CA, USA(加州大学戴维斯分校放射学系)