面向推理的提示优化以对齐黑盒大语言模型
Inference-Aware Prompt Optimization for Aligning Black-Box Large Language Models
AI总结：
本文提出IAPO框架，通过联合优化提示和推理规模，提升黑盒大语言模型对齐效果。
摘要：
提示优化方法在对齐黑盒大语言模型（LLMs）方面已显示出显著的有效性。同时，推理扩展策略如Best-of-N采样和多数投票也已显示通过用额外计算换取更好输出来改善对齐和性能。然而，现有的提示优化方法是推理策略无关的；也就是说，它们优化提示而不考虑推理策略。这构成了一个显著的方法论缺口，因为我们实证和理论分析揭示了这两个范式之间有强烈的相互依赖性。此外，我们发现用户在多个目标之间的权衡和推理预算方面的偏好显著影响提示和推理配置的选择。为了解决这一缺口，我们引入了一个新的统一框架，称为IAPO（面向推理的提示优化），它联合优化提示和推理规模，同时考虑到推理预算和不同任务目标。然后，我们为IAPO开发了一个固定预算训练算法，称为PSST（通过顺序修剪的提示扩展），并建立了有限预算下的误差概率保证。最后，我们在六个任务上评估了PSST的有效性，包括多目标文本生成和推理，并展示了在提示优化中纳入推理意识在对齐黑盒LLMs中的关键作用。
英文摘要：
Prompt optimization methods have demonstrated significant effectiveness in aligning black-box large language models (LLMs). In parallel, inference scaling strategies such as Best-of-N Sampling and Majority Voting have likewise been shown to improve alignment and performance by trading additional computation for better output. However, existing prompt optimization approaches are inference strategy agnostic; that is, they optimize prompts without accounting for the inference strategy. This constitutes a significant methodological gap, as our empirical and theoretical analysis reveals a strong interdependence between these two paradigms. Moreover, we find that user preferences regarding trade-offs among multiple objectives and inference budgets substantially influence the choice of prompt and inference configuration. To address this gap, we introduce a novel unified framework named IAPO (Inference-Aware Prompt Optimization) that jointly optimizes the prompt and inference scale, while being aware of the inference budget and different task objectives. We then develop a fixed-budget training algorithm for IAPO, called PSST (Prompt Scaling via Sequential Trimming), and establish finite-budget guarantees on the error probability. Finally, we evaluate the effectiveness of PSST on six tasks, including multi-objective text generation and reasoning, and demonstrate the critical role of incorporating inference-awareness in aligning black-box LLMs using prompt optimization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
