更容易判断而非寻找:预测上下文学习成功以选择演示
Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection
AI总结:
本文提出DiSP框架,通过分层判断和轻量路由模型,预测上下文学习中演示选择的成功率,从而在多个数据集上实现更高的准确率和更快的推理速度。
摘要:
上下文学习(ICL)对提示中出现的演示非常敏感,但选择演示是昂贵的,因为可能的演示上下文和组合空间极大。我们主张演示选择是'更容易判断而非寻找':预测特定查询-上下文对(q,D)是否成功比搜索最优D*更便宜且更通用。基于这一见解,我们提出DiSP,一种样本和判断框架,通过分层查询难度进行分类。DiSP运行随机演示试验以估计每个训练查询的成功率,训练轻量级路由器预测查询难度,并训练针对特定层次的判断器对采样演示进行判断。在推理时,DiSP在显式预算下执行停止接受判断,当未找到合适上下文时会发出诊断风险标签。在五个分类数据集上使用Llama 3-8B和Qwen 2.5-7B,DiSP实现了最佳的平均准确率,比强学习选择基线提高了最高3.4%,同时实现了高达23倍的端到端时间加速。
英文摘要:
In-context learning (ICL) is highly sensitive to which demonstrations appear in the prompt, but selecting them is expensive because the space of possible demonstration contexts and combinations is enormous. We argue that demonstration selection is \emph{easier to judge than to find}: predicting whether a specific query--context pair $(q,D)$ will succeed is cheaper and more general than searching for an optimal $D^\star$. Based on this insight, we propose DiSP, a sample-and-judge framework that stratifies queries by difficulty. DiSP runs random demonstration trials to estimate success rate of each training query, trains a lightweight router to predict difficulty from the query, and trains level-specific judges for sampled demonstrations. At inference, DiSP performs stop-on-acceptance judging under an explicit budget, emitting diagnostic risk tags when no suitable context is found. Across five classification datasets with Llama~3--8B and Qwen~2.5--7B, DiSP achieves the best average accuracy, improving over strong learned selection baselines by up to 3.4\%, while achieving up to $23\times$ end-to-end wall-clock speedup.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。