一种统一评估指导的查询相关提示优化框架
Knowing How to Edit: Reliable Evaluation Signals for Diagnosing and Optimizing Prompts at Query Level
AI总结：
提出一个基于性能导向的提示评估框架，并开发一个无需执行的评估器来预测多维质量分数，进而指导一个度量感知优化器以可解释的查询相关方式重写提示，在多个数据集和骨干模型上优于现有方法。
摘要：
大多数提示优化方法优化单个静态模板，使其在复杂和动态的用户场景中无效。现有的查询相关方法依赖于不稳定的文本反馈或黑盒奖励模型，提供弱且不可解释的优化信号。更根本的是，提示质量本身缺乏统一、系统的定义，导致碎片化和不可靠的评估信号。我们的方法首先建立了一个面向性能的、系统的、全面的提示评估框架。此外，我们开发并微调了一个无需执行的评估器，可以直接从文本中预测多维质量分数。然后，评估器指导一个度量感知优化器，该优化器以可解释的、查询相关的方式诊断失败模式并重写提示。我们的评估器在预测提示性能方面达到了最强的准确性，并且评估指导的优化在八个数据集和三个骨干模型上始终优于静态模板和查询相关的基线。总的来说，我们提出了一个统一的、基于度量的提示质量视角，并证明了我们的评估指导优化流程在多样化任务中提供了稳定、可解释和模型无关的改进。
英文摘要：
Prompt optimization has become a central mechanism for eliciting strong performance from LLMs, and recent work has made substantial progress by proposing diverse prompt evaluation metrics and optimization strategies. Despite these advances, prompt evaluation and prompt optimization are often developed in isolation, limiting the extent to which evaluation can effectively inform prompt refinement. In this work, we study prompt optimization as a process guided by performance-relevant evaluation signals. To address the disconnect between evaluation and optimization, we propose an evaluation-instructed prompt optimization approach that explicitly connects prompt evaluation with query-dependent optimization. Our method integrates multiple complementary prompt quality metrics into a performance-reflective evaluation framework and trains an execution-free evaluator that predicts prompt quality directly from text, avoiding repeated model executions. These evaluation signals then guide prompt refinement in a targeted and interpretable manner. Empirically, the proposed evaluator achieves 83.7% accuracy in predicting prompt performance. When incorporated into the optimization process, our approach consistently outperforms existing optimization baselines across eight benchmark datasets and three different backbone LLMs. Overall, our results demonstrate that reliable and efficient evaluation signals can serve as an effective foundation for robust and interpretable prompt optimization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
