蒙特卡洛采样用于分析上下文示例
Monte Carlo Sampling for Analyzing In-Context Examples
AI总结:
本研究提出蒙特卡洛采样方法,分析上下文示例数量、顺序与选择间的相互作用,发现先前指导不普适,且基于数据估值的示例选择虽鲁棒却性能意外下降。
摘要:
先前的研究表明,上下文学习对呈现因素(如所选示例的顺序、数量和选择)具有脆弱性。然而,基于消融的关于选择示例数量的指导可能忽略了不同呈现因素之间的相互作用。在本工作中,我们开发了一种基于蒙特卡洛采样的方法,在研究示例数量影响的同时,明确考虑顺序和所选示例的影响。我们发现,先前关于应选择多少上下文示例的指导并不总是适用于不同的所选示例集合和顺序,并且一次性设置是否优于零次设置高度依赖于所选示例。此外,受数据估值启发,我们将我们的采样方法应用于上下文示例选择,以选择在不同顺序下表现良好的示例。我们发现了一个负面结果,即虽然性能对顺序和示例数量具有鲁棒性,但与随机采样相比,存在意外的性能下降。
英文摘要:
Prior works have shown that in-context learning is brittle to presentation factors such as the order, number, and choice of selected examples. However, ablation-based guidance on selecting the number of examples may ignore the interplay between different presentation factors. In this work we develop a Monte Carlo sampling-based method to study the impact of number of examples while explicitly accounting for effects from order and selected examples. We find that previous guidance on how many in-context examples to select does not always generalize across different sets of selected examples and orderings, and whether one-shot settings outperform zero-shot settings is highly dependent on the selected example. Additionally, inspired by data valuation, we apply our sampling method to in-context example selection to select examples that perform well across different orderings. We find a negative result, that while performance is robust to ordering and number of examples, there is an unexpected performance degradation compared to random sampling.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。