In-context learning with large language models (LLMs) excels at adapting to various tasks rapidly. However, its success hinges on carefully selecting demonstrations, which remains an obstacle in practice. Current approaches to this problem either rely on hard-to-acquire external supervision or require frequent interactions with LLMs, resulting in high costs. We propose a new method called In-Context Reflection (ICR) to overcome these challenges. ICR strategically selects demonstrations to reduce the discrepancy between the LLM's outputs and the actual input-output mappings. Specifically, ICR starts with a random set of initial demonstrations, then iteratively refines it. In each step, it analyzes a pool of candidate examples and identifies the ones most likely to challenge the LLM's current understanding, measured by a new metric called misconfidence. These most confusing examples are then selected to replace the less informative demonstrations in the current set. Our comprehensive evaluation across five diverse datasets encompassing 13 subtasks shows the efficacy of ICR. Compared to existing methods, ICR achieves an average performance boost of 4%, while demonstrating remarkable cross-task generalization capabilities.
Revisiting Demonstration Selection Strategies in In-Context Learning
Beihang University(北京航空航天大学);The University of Sydney(悉尼大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Nanyang Technological University(南洋理工大学);The Hong Kong Polytechnic University(香港理工大学)
UniICL: An Efficient Unified Framework Unifying Compression, Selection, and Generation
School of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Hong Kong Polytechnic University(香港理工大学);Stepfun
Selecting Demonstrations for Many-Shot In-Context Learning via Gradient Matching
School of Computer Science and Engineering, Beihang University, China(北京航空航天大学计算机科学与工程学院);Meituan, Inc.(美团公司);Beijing Institute for General Artificial Intelligence, China(北京通用人工智能研究院);Ping An Technology, China(平安科技);Department of Computer Science, University of Manchester, UK(曼彻斯特大学计算机科学系)
Delta-KNN: Improving Demonstration Selection in In-Context Learning for Alzheimer's Disease Detection
Department of Computer Science(计算机科学系);Vancouver Stroke Program and Division of Neurology, Faculty of Medicine(温哥华卒中计划和神经病学系,医学院);The University of British Columbia(不列颠哥伦比亚大学)
Meta-Sel: Efficient Demonstration Selection for In-Context Learning via Supervised Meta-Learning
BNU-BNBU Institute of Artificial Intelligence(BNU-BNBU人工智能研究院);Future Networks, Beijing Normal-Hong Kong Baptist University(未来网络,北京师范大学-香港 Baptist大学);Beijing Normal University at Zhuhai(珠海北京师范大学)
Iterative Forward Tuning Boosts In-Context Learning in Language Models
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Alibaba Group(阿里巴巴集团);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);Shanghai AI Laboratory(上海人工智能实验室)
DemoShapley: Valuation of Demonstrations for In-Context Learning
Department of Mathematics and Computer Science(数学与计算机科学系);Eindhoven University of Technology(埃因霍温理工大学);Intel Lab(英特尔实验室);Department of Psychology(心理学系);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Department of Data Science(数据科学系);New Jersey Institute of Technology(新泽西理工学院);Department of Computer Science(计算机科学系);University of Illinois Chicago(伊利诺伊大学芝加哥分校)
Exploring Imbalanced Annotations for Effective In-Context Learning
Department of Statistics and Data Science, Southern University of Science and Technology(南方科技大学统计与数据科学系);School of Economics and Finance, Xi’an Jiaotong University(西安交通大学经济金融学院);School of Mathematics and Statistics, Xi’an Jiaotong University(西安交通大学数学与统计学学院)