In-context learning (ICL) is a few-shot learning paradigm that involves learning mappings through input-output pairs and appropriately applying them to new instances. Despite the remarkable ICL capabilities demonstrated by Large Language Models (LLMs), existing works are highly dependent on large-scale labeled support sets, not always feasible in practical scenarios. To refine this approach, we focus primarily on an innovative selective annotation mechanism, which precedes the standard demonstration retrieval. We introduce the Language Model-based Determinant Point Process (LM-DPP) that simultaneously considers the uncertainty and diversity of unlabeled instances for optimal selection. Consequently, this yields a subset for annotation that strikes a trade-off between the two factors. We apply LM-DPP to various language models, including GPT-J, LlaMA, and GPT-3. Experimental results on 9 NLU and 2 Generation datasets demonstrate that LM-DPP can effectively select canonical examples. Further analysis reveals that LLMs benefit most significantly from subsets that are both low uncertainty and high diversity.
Exploring Imbalanced Annotations for Effective In-Context Learning
Department of Statistics and Data Science, Southern University of Science and Technology（南方科技大学统计与数据科学系）；School of Economics and Finance, Xi’an Jiaotong University（西安交通大学经济金融学院）；School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学学院）
Selective Annotation Makes Language Models Better Few-Shot Learners
The University of Hong Kong（香港大学）；University of Washington（华盛顿大学）；Carnegie Mellon University（卡内基梅隆大学）；Meta AI；Penn State University（宾夕法尼亚州立大学）；Allen Institute for AI（艾伦人工智能研究所）
Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ZJU-UIUC Institute, Zhejiang University（浙江大学ZJU-UIUC学院）；National University of Singapore（新加坡国立大学）
An Open Natural Language Processing Development Framework for EHR-based Clinical Research: A case demonstration using the National COVID Cohort Collaborative (N3C)
Mayo Clinic（妙佑医疗国际）；Tufts Clinical and Translational Science Institute（塔夫茨临床与转化科学研究所）；Tufts Medical Center（塔夫茨医疗中心）；University of Kentucky（肯塔基大学）；University of Kansas Medical Center（堪萨斯大学医学中心）；University of Minnesota at Twin Cities（明尼苏达大学双城分校）；University of Florida（佛罗里达大学）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；University of Michigan Medical School（密歇根大学医学院）；Wake Forest Baptist Medical Center（维克森林浸信会医疗中心）；Johns Hopkins University（约翰斯·霍普金斯大学）；Stony Brook University（石溪大学）；Sage Bionetwork（Sage生物网络）
虽然我们关注临床自然语言处理（NLP）的最新进展，但我们可以注意到，由于透明度、可解释性和可用性有限，临床和转化研究社区对采用NLP模型存在一些阻力。在这项研究中，我们提出了一个开放的自然语言处理开发框架。我们通过为国家COVID队列协作（N3 C）实施NLP算法对其进行了评估。基于对从COVID-19相关临床记录中提取信息抽取的兴趣，我们的工作包括1）使用COVID-19体征和症状作为用例的开放数据注释过程，2）社区驱动的规则组成平台，以及3）合成文本数据生成工作流程，以生成用于信息抽取任务的文本，而不涉及人类受试者。语料库来自三个不同的机构（梅奥诊所，肯塔基大学，明尼苏达大学）的文本。金标准注释与一个单一的机构（梅奥）的规则进行了测试。这导致Mayo、Minnesota和Kentucky测试数据集的F分数分别为0.876、0.706和0.694。该研究作为N3 C NLP子组的联盟努力，证明了创建联合NLP算法开发和基准测试平台以加强多机构临床NLP研究和采用的可行性。虽然我们在这项工作中使用COVID-19作为用例，但我们的框架足够通用，可以应用于临床NLP的其他感兴趣领域。
