教得更好还是展示得更聪明？关于自动提示优化中的指令与示例
Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimization
AI总结：
本研究系统比较了自动提示优化中的指令优化与示例优化方法，发现示例选择策略（如随机搜索）可超越先进指令优化，且两者组合具有协同效应，强调示例优化在未来研究中应受更多重视。
摘要：
大型语言模型已展现出卓越的能力，但其性能在很大程度上依赖于有效的提示工程。自动提示优化（APO）方法旨在自动化这一过程，并大致可分为针对指令优化的方法（指令优化，IO）和针对示例优化的方法（示例优化，EO）。尽管它们目标相同，但这两类方法却相对独立地发展，其中IO近年来获得了更多的研究关注。本文旨在弥合这一差距，通过在多样化的具有挑战性的任务集上，全面比较代表性IO和EO技术在单独使用及组合使用时的性能。我们的研究结果表明，智能地重用从验证集上评估提示时生成的模型输入-输出对作为示例，能够持续提升IO方法的性能，但这一方向目前研究不足。我们还发现，尽管近期研究聚焦于IO，但示例的选择方式可能比指令优化方式更为重要，即便是随机搜索这样简单的EO策略，也能在无需任何优化的情况下，超越使用种子指令的最先进IO方法。此外，我们观察到EO与IO之间存在协同效应，最优组合的性能超越了各自单独贡献的总和。我们得出结论，将示例优化作为独立方法研究，以及探索其与指令优化的最优组合，仍是APO的关键方面，值得在未来研究中给予更多关注，即使在指令跟随能力高度强大的模型时代也是如此。
英文摘要：
Large language models have demonstrated remarkable capabilities, but their performance is heavily reliant on effective prompt engineering. Automatic prompt optimization (APO) methods are designed to automate this and can be broadly categorized into those targeting instructions (instruction optimization, IO) vs. those targeting exemplars (exemplar optimization, EO). Despite their shared objective, these have evolved rather independently, with IO receiving more research attention recently. This paper seeks to bridge this gap by comprehensively comparing the performance of representative IO and EO techniques both isolation and combination on a diverse set of challenging tasks. Our findings reveal that intelligently reusing model-generated input-output pairs obtained from evaluating prompts on the validation set as exemplars, consistently improves performance on top of IO methods but is currently under-investigated. We also find that despite the recent focus on IO, how we select exemplars can outweigh how we optimize instructions, with EO strategies as simple as random search outperforming state-of-the-art IO methods with seed instructions without any optimization. Moreover, we observe a synergy between EO and IO, with optimal combinations surpassing the individual contributions. We conclude that studying exemplar optimization both as a standalone method and its optimal combination with instruction optimization remain a crucial aspect of APO and deserve greater consideration in future research, even in the era of highly capable instruction-following models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
