Large language models (LLMs) can use in-context demonstrations to improve performance on zero-shot tasks. However, selecting the best in-context examples is challenging because model performance can vary widely depending on the selected examples. We present a cross-entropy difference (CED) method for selecting in-context demonstrations. Our method is based on the observation that the effectiveness of in-context demonstrations negatively correlates with the perplexity of the test example by a language model that was finetuned on that demonstration. We utilize parameter efficient finetuning to train small models on training data that are used for computing the cross-entropy difference between a test example and every candidate in-context demonstration. This metric is used to rank and select in-context demonstrations independently for each test input. We evaluate our method on a mix-domain dataset that combines 8 benchmarks, representing 4 text generation tasks, showing that CED for in-context demonstration selection can improve performance for a variety of LLMs.
Meta-Sel: Efficient Demonstration Selection for In-Context Learning via Supervised Meta-Learning
BNU-BNBU Institute of Artificial Intelligence(BNU-BNBU人工智能研究院);Future Networks, Beijing Normal-Hong Kong Baptist University(未来网络,北京师范大学-香港 Baptist大学);Beijing Normal University at Zhuhai(珠海北京师范大学)
Revisiting Demonstration Selection Strategies in In-Context Learning
Beihang University(北京航空航天大学);The University of Sydney(悉尼大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Nanyang Technological University(南洋理工大学);The Hong Kong Polytechnic University(香港理工大学)
UniICL: An Efficient Unified Framework Unifying Compression, Selection, and Generation
School of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Hong Kong Polytechnic University(香港理工大学);Stepfun
Selecting Demonstrations for Many-Shot In-Context Learning via Gradient Matching
School of Computer Science and Engineering, Beihang University, China(北京航空航天大学计算机科学与工程学院);Meituan, Inc.(美团公司);Beijing Institute for General Artificial Intelligence, China(北京通用人工智能研究院);Ping An Technology, China(平安科技);Department of Computer Science, University of Manchester, UK(曼彻斯特大学计算机科学系)
Delta-KNN: Improving Demonstration Selection in In-Context Learning for Alzheimer's Disease Detection
Department of Computer Science(计算机科学系);Vancouver Stroke Program and Division of Neurology, Faculty of Medicine(温哥华卒中计划和神经病学系,医学院);The University of British Columbia(不列颠哥伦比亚大学)
DemoShapley: Valuation of Demonstrations for In-Context Learning
Department of Mathematics and Computer Science(数学与计算机科学系);Eindhoven University of Technology(埃因霍温理工大学);Intel Lab(英特尔实验室);Department of Psychology(心理学系);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Department of Data Science(数据科学系);New Jersey Institute of Technology(新泽西理工学院);Department of Computer Science(计算机科学系);University of Illinois Chicago(伊利诺伊大学芝加哥分校)
Learning to Select In-Context Demonstration Preferred by Large Language Model
School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);Microsoft Research Asia(微软亚洲研究院);State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China(一般人工智能国家重点实验室)
GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
School of Mathematical Sciences, Shanghai Jiao Tong University(上海交通大学数学科学学院);Institute of Natural Sciences, Shanghai Jiao Tong University(上海交通大学自然科学研究院)
Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models
University of Trieste(的里雅斯特大学);University of Milano-Bicocca(米兰-比科卡大学);University of Groningen(格罗宁根大学);Khoury College of Computer Sciences, Northeastern University(东北大学科里计算机学院)
School of Computer and Communication Engineering, University of Science and Technology Beijing(计算机与通信工程学院,科学技术大学);National Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence(通用人工智能国家重点实验室,北京通用人工智能研究院);Shunde Innovation School, University of Science and Technology Beijing(顺德创新学院,科学技术大学)
Exploring Imbalanced Annotations for Effective In-Context Learning
Department of Statistics and Data Science, Southern University of Science and Technology(南方科技大学统计与数据科学系);School of Economics and Finance, Xi’an Jiaotong University(西安交通大学经济金融学院);School of Mathematics and Statistics, Xi’an Jiaotong University(西安交通大学数学与统计学学院)
Iterative Forward Tuning Boosts In-Context Learning in Language Models
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Alibaba Group(阿里巴巴集团);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);Shanghai AI Laboratory(上海人工智能实验室)
LODESTAR: Robust Entropy-Based Answer Selection in Retrieval-Augmented Generation for Question Answering -- Directing Frozen-LLM Entropy with a Reinforcement-Learned Prompt Polarizer under Misleading Passages