DemoShapley:为上下文学习评估演示
DemoShapley: Valuation of Demonstrations for In-Context Learning
AI总结:
DemoShapley通过Shapley值评估演示贡献,Beta-DemoShapley优化低样本场景,提升ICL演示评估的鲁棒性和性能。
摘要:
大型语言模型(LLMs)利用上下文学习(ICL)在许多任务中无需特定任务微调即可表现出色。然而,演示选择和排序对ICL效果影响显著。针对此问题,我们提出了DemoShapley,一种基于Shapley值的方法,通过测量不同提示排列中的边际效应来评估每个演示的贡献。为进一步考虑ICL的有限上下文窗口和频繁的低样本设置,我们引入了Beta-DemoShapley,一种加权扩展方法,强调较小提示大小的影响。在多个基准测试中,DemoShapley一致优于现有影响基选择策略,而Beta-DemoShapley在低样本场景中进一步提升性能。两种方法还能够检测错误标记的数据,增强对分布外任务的泛化能力,并减少人口偏差。共同来看,它们为ICL中的演示评估提供了统一且稳健的框架。
英文摘要:
Large language models (LLMs) using in-context learning (ICL) excel in many tasks without task-specific fine-tuning. However, demonstration selection and ordering greatly impact ICL effectiveness. Focus on this issue, we propose DemoShapley, a Shapley-value based method that evaluates each demonstration's contribution by measuring its marginal effect across different prompt permutations. To further account for ICL's limited context windows and frequent low-shot settings, we introduce Beta-DemoShapley, a weighted extension that emphasizes the influence of smaller prompt sizes. Experiments on multiple benchmarks show that DemoShapley consistently outperforms existing influence-based selection strategies, while Beta-DemoShapley further improves performance in low-shot scenarios. Both methods also detect mislabeled data, enhance generalization to out-of-distribution tasks, and reduce demographic bias. Together, they provide a unified and robust framework for demonstration valuation in ICL.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。