PEEM:提示工程评估指标用于可解释地联合评估提示和响应
PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses
AI总结:
本文提出PEEM,一种联合且可解释的提示和响应评估框架,通过9个轴线评估提示和响应质量,结合LLM评估器输出评分和解释,提升提示设计的可解释性和优化效果。
摘要:
提示设计是大型语言模型(LLMs)的主要控制接口,但标准评估往往仅关注回答正确性,掩盖了提示成功或失败的原因,并缺乏指导性。本文提出PEEM(提示工程评估指标),一种统一框架,用于联合和可解释地评估提示和响应。PEEM定义了9个轴线:3个提示标准(清晰度/结构、语言质量、公平性)和6个响应标准(准确性、连贯性、相关性、客观性、清晰度、简洁性),并使用基于LLM的评估器输出(i)1-5分的标量评分和(ii)基于标准的自然语言解释。在7个基准和5个任务模型上,PEEM的准确性轴与传统准确性高度一致,同时保持模型排名(汇总斯皮尔曼rho约0.97,皮尔逊r约0.94,p < 0.001)。多评估者研究显示一致的相对判断(成对rho = 0.68-0.85),支持评估者无关的部署。除了对齐,PEEM捕捉互补的语言失败模式并在提示扰动下保持信息:提示质量趋势跟踪下游准确性在迭代重写下的变化,语义对抗性操纵导致清晰的评分退化,而意义保持的改写产生高稳定性(鲁棒率约76.7-80.6%)。最后,仅使用PEEM评分和解释作为反馈,零样本提示重写循环将下游准确性提高11.7个百分点,优于监督和基于强化学习的提示优化基线。总体而言,PEEM提供了一种可重复、基于标准的协议,将提示制定与响应行为联系起来,并启系统诊断和优化LLM交互。
英文摘要:
Prompt design is a primary control interface for large language models (LLMs), yet standard evaluations largely reduce performance to answer correctness, obscuring why a prompt succeeds or fails and providing little actionable guidance. We propose PEEM (Prompt Engineering Evaluation Metrics), a unified framework for joint and interpretable evaluation of both prompts and responses. PEEM defines a structured rubric with 9 axes: 3 prompt criteria (clarity/structure, linguistic quality, fairness) and 6 response criteria (accuracy, coherence, relevance, objectivity, clarity, conciseness), and uses an LLM-based evaluator to output (i) scalar scores on a 1-5 Likert scale and (ii) criterion-specific natural-language rationales grounded in the rubric. Across 7 benchmarks and 5 task models, PEEM's accuracy axis strongly aligns with conventional accuracy while preserving model rankings (aggregate Spearman rho about 0.97, Pearson r about 0.94, p < 0.001). A multi-evaluator study with four models shows consistent relative judgments (pairwise rho = 0.68-0.85), supporting evaluator-agnostic deployment. Beyond alignment, PEEM captures complementary linguistic failure modes and remains informative under prompt perturbations: prompt-quality trends track downstream accuracy under iterative rewrites, semantic adversarial manipulations induce clear score degradation, and meaning-preserving paraphrases yield high stability (robustness rate about 76.7-80.6%). Finally, using only PEEM scores and rationales as feedback, a zero-shot prompt rewriting loop improves downstream accuracy by up to 11.7 points, outperforming supervised and RL-based prompt-optimization baselines. Overall, PEEM provides a reproducible, criterion-driven protocol that links prompt formulation to response behavior and enables systematic diagnosis and optimization of LLM interactions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。