Selective Annotation Makes Language Models Better Few-Shot Learners
The University of Hong Kong（香港大学）；University of Washington（华盛顿大学）；Carnegie Mellon University（卡内基梅隆大学）；Meta AI；Penn State University（宾夕法尼亚州立大学）；Allen Institute for AI（艾伦人工智能研究所）
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, Tao Yu
Many recent approaches to natural language tasks are built on the remarkable abilities of large language models. Large language models can perform in-context learning, where they learn a new task from a few task demonstrations, without any parameter updates. This work examines the implications of in-context learning for the creation of datasets for new natural language tasks. Departing from recent in-context learning methods, we formulate an annotation-efficient, two-step framework: selective annotation that chooses a pool of examples to annotate from unlabeled data in advance, followed by prompt retrieval that retrieves task examples from the annotated pool at test time. Based on this framework, we propose an unsupervised, graph-based selective annotation method, voke-k, to select diverse, representative examples to annotate. Extensive experiments on 10 datasets (covering classification, commonsense reasoning, dialogue, and text/code generation) demonstrate that our selective annotation method improves the task performance by a large margin. On average, vote-k achieves a 12.9%/11.4% relative gain under an annotation budget of 18/100, as compared to randomly selecting examples to annotate. Compared to state-of-the-art supervised finetuning approaches, it yields similar performance with 10-100x less annotation cost across 10 tasks. We further analyze the effectiveness of our framework in various scenarios: language models with varying sizes, alternative selective annotation methods, and cases where there is a test data domain shift. We hope that our studies will serve as a basis for data annotations as large language models are increasingly applied to new tasks. Our code is available at https://github.com/HKUNLP/icl-selective-annotation.
Sub-SA: Strengthen In-context Learning via Submodular Selective Annotation
School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；School of Automation, Southeast University（东南大学自动化学院）；State Key Laboratory of ASIC and System, Fudan University（复旦大学专用集成电路与系统国家重点实验室）
YACLC: A Chinese Learner Corpus with Multidimensional Annotation
Beijing Language and Culture University（北京语言大学）；Beijing Normal University（北京师范大学）；Yunnan Normal University（云南师范大学）；Northeastern University（东北大学）；Shanghai University of Finance and Economics（上海财经大学）；Tsinghua University（清华大学）
学习者语料库收集第二语言学习者，即第二语言或外语学习者的语言数据。该资源对第二语言习得研究、外语教学和自动语法纠错具有重要意义。然而，针对对外汉语学习者的学习者语料库研究却很少。因此，我们建议建立一个大规模、多维的汉语学习者语料库。为了构建语料库，我们首先获得了大量的由汉语学习者生成的富文本。然后，我们设计了一个注释方案，包括句子的可接受性评分以及语法错误和流畅性为基础的更正。我们建立了一个众包平台来有效地执行注释（yaclc.wenmind.net）。我们将该语料库命名为YACLC（Yet Another Chinese Learner Corpus），并将其作为CUGE基准测试的一部分（http：//www.example.com）发布。cuge.baai.ac.cn通过对语料库中的原文和注释的分析，我们发现YACLC具有相当大的规模和非常高的注释质量。希望该语料库能进一步促进汉语国际教育和汉语语法错误自动纠正的研究。
Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models
Harbin Institute of Technology（哈尔滨工业大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Unlimidata Ltd（Unlimidata有限公司）；The Chinese University of Hong Kong（香港中文大学）；University of Calabria（卡拉布里亚大学）
Annotating Errors in English Learners' Written Language Production: Advancing Automated Written Feedback Systems
Tohoku University, Sendai, Japan（东京东京大学）；RIKEN, Wako, Japan（日本理化学研究所）；ALTA Institute, Computer Laboratory, University of Cambridge, U.K（剑桥大学计算机实验室ALTA研究所）；INSA Lyon, Villeurbanne, France（里昂国立应用科学学院）；CNRS, LIS, Aix-Marseille University, Marseille, France（国家科学研究中心（CNRS））；MBZUAI, Abu Dhabi, United Arab Emirates（阿布扎赫德人工智能研究所）
IntrEx: A Dataset for Modeling Engagement in Educational Conversations
Department of Computer Science, University of Warwick, UK（沃里克大学计算机科学系）；School of Computer Science, University of Sheffield, UK（谢菲尔德大学计算机科学学院）；Department of Psychology, University of Warwick, UK（沃里克大学心理学系）
To Err Is Human; To Annotate, SILICON? Toward Robust Reproducibility in LLM Annotation
Robert H. Smith School of Business, University of Maryland（马里兰大学罗伯特·H·史密斯商学院）；Leonard N. Stern School of Business, New York University（纽约大学伦纳德·N·斯坦商学院）
Making Language Models Better Tool Learners with Execution Feedback
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ZJU-Ant Group Joint Research Center for Knowledge Graphs, Zhejiang University（浙江大学-蚂蚁集团知识图谱联合研究中心）；ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University（浙江大学杭州国际科技创新中心）；Alibaba Group（阿里巴巴集团）
Understanding Learner-LLM Chatbot Interactions and the Impact of Prompting Guidelines
Università degli Studi di Milano Bicocca（米兰比可卡大学）；Universitat Pompeu Fabra（庞培法布拉大学）；University of Regensburg（雷根斯堡大学）；Politecnico di Milano（米兰理工大学）；Istituto per le Tecnologie Didattiche (ITD-CNR)（国家研究委员会教育技术研究所）；Università degli Studi di Milano（米兰大学）；Institute of Science and Technology of Cognition (ISTC-CNR)（国家研究委员会认知科学与技术研究所）
RadAnnotate: Large Language Models for Efficient and Reliable Radiology Report Annotation
Department of Computer Science, University of California, Davis, CA, USA（加州大学戴维斯分校计算机科学系）；Department of Radiology, University of California, Davis, CA, USA（加州大学戴维斯分校放射学系）
XAL: EXplainable Active Learning Makes Classifiers Better Low-resource Learners
School of Engineering, Westlake University（西湖大学工程学院）；WeChat AI, Tencent Inc.（腾讯微信人工智能）；Zhejiang University School of Cyber Science and Engineering, Sichuan University（四川大学浙江大学网络空间安全学院）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖高等研究院先进技术研究所）
