通过强化学习进行上下文学习的演示选择
Demonstration Selection for In-Context Learning via Reinforcement Learning
AI总结：
本文提出RDES方法，利用强化学习（Q-learning和PPO）优化上下文学习中的演示选择，平衡多样性与相关性，在14个LLM和多个基准上超越十个基线，并融入CoT进一步提升性能。
摘要：
演示选择的多样性对于通过更广泛覆盖结构和概念来增强模型泛化能力至关重要。构建合适的演示集仍是一个关键的研究挑战。本文介绍了相关性-多样性增强选择（RDES），这是一种创新方法，利用强化学习（RL）框架来优化为适用于上下文学习（ICL）的任务（特别是文本分类和推理）在少样本提示场景中选择多样化的参考演示。RDES采用Q学习和基于PPO的变体等框架，动态识别能够最大化多样性（通过标签分布量化）和与任务目标相关性的演示。该策略确保参考数据的均衡表示，从而提高准确性和泛化能力。通过在多个基准数据集（包括多样化的推理任务）上进行的广泛实验，涉及14个闭源和开源大语言模型（LLM），我们证明RDES相比十个已建立的基线显著提升了性能。我们的评估包括在选定数据集上对不同演示数量的性能分析。此外，我们研究了融入链式思维（CoT）推理，这进一步提升了预测性能。结果凸显了强化学习在自适应演示选择和解决ICL挑战方面的潜力。
英文摘要：
Diversity in demonstration selection is critical for enhancing model generalization by enabling broader coverage of structures and concepts. Constructing appropriate demonstration sets remains a key research challenge. This paper introduces the Relevance-Diversity Enhanced Selection (RDES), an innovative approach that leverages reinforcement learning (RL) frameworks to optimize the selection of diverse reference demonstrations for tasks amenable to in-context learning (ICL), particularly text classification and reasoning, in few-shot prompting scenarios. RDES employs frameworks like Q-learning and a PPO-based variant to dynamically identify demonstrations that maximize both diversity (quantified by label distribution) and relevance to the task objective. This strategy ensures a balanced representation of reference data, leading to improved accuracy and generalization. Through extensive experiments on multiple benchmark datasets, including diverse reasoning tasks, and involving 14 closed-source and open-source LLMs, we demonstrate that RDES significantly enhances performance compared to ten established baselines. Our evaluation includes analysis of performance across varying numbers of demonstrations on selected datasets. Furthermore, we investigate incorporating Chain-of-Thought (CoT) reasoning, which further boosts predictive performance. The results highlight the potential of RL for adaptive demonstration selection and addressing challenges in ICL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
