探索不平衡标注以实现有效的上下文学习
Exploring Imbalanced Annotations for Effective In-Context Learning
AI总结：
针对标注数据集类别不平衡导致上下文学习性能下降的问题，提出条件偏差重加权方法RCB，结合类别权重与条件偏差对示例得分重新加权，将现有选择方法的平均准确率最高提升5.42%。
摘要：
大型语言模型（LLMs）通过上下文学习（ICL）在下游任务中展现出卓越性能，该学习方式严重依赖于从标注数据集中选取的示例。然而，在真实场景中，这些数据集常呈现长尾类别分布，导致示例选择产生偏差。本研究表明，无论采用何种选择方法，此类类别不平衡均会显著降低跨多种任务的ICL性能。此外，仅关注类别权重的经典重平衡方法因忽视条件偏差（即类内偏斜的特征分布）而导致性能不佳。为解决此问题，我们提出了一种简单且互补的方法——条件偏差重加权（简称RCB），以提升类别不平衡下的ICL性能。具体而言，RCB利用均衡子集估计条件偏差，并结合类别权重与条件偏差对示例得分进行重新加权。由此，RCB在防止过度选取主导类示例的同时，保留了现有选择方法的有效性。在通用基准上的大量实验证明了本方法的有效性，将现有选择方法的平均准确率最高提升了5.42%。
英文摘要：
Large language models (LLMs) have shown impressive performance on downstream tasks through in-context learning (ICL), which heavily relies on the demonstrations selected from annotated datasets. However, these datasets often exhibit long-tailed class distributions in real-world scenarios, leading to biased demonstration selection. In this work, we show that such class imbalances significantly degrade the ICL performance across various tasks, regardless of selection methods. Moreover, classical rebalancing methods, which focus solely on class weights, yield poor performance due to neglecting condition bias--skewed feature distributions within classes. To address this, we propose Reweighting with Conditional Bias (dubbed RCB), a simple and complementary approach to enhance ICL performance under class imbalance. In particular, RCB estimates conditional bias using a balanced subset and re-weights demonstration scores based on both class weight and conditional bias. In effect, RCB prevents over-selection from dominant classes while preserving the efficacy of current selection methods. Extensive experiments on common benchmarks demonstrate the effectiveness of our method, improving the average accuracy of current selection methods by up to 5.42%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
