GLaPE：大语言模型的无金标提示词评估与优化
GLaPE: Gold Label-agnostic Prompt Evaluation and Optimization for Large Language Model
AI总结：
针对现有LLM提示词优化依赖人工金标评估的局限，提出无金标提示词评估方法GLaPE，以自一致性为初始评分并优化相同答案提示词的分数，在无金标时仍能提供可靠评估，且在六个推理任务中优化效果与基于准确率的方法相当。
摘要：
尽管大语言模型（LLMs）发展迅速，但其任务性能仍对提示词设计敏感。近期研究探索利用LLM自身作为优化器，以识别最大化任务准确率的最优提示词。然而，在评估提示词时，这类方法严重依赖难以获取的人工标注金标来计算每个候选提示词的任务准确率，这阻碍了其广泛应用和通用性。为克服这一局限，本研究提出无金标提示词评估（GLaPE）以减轻对金标的依赖。基于观察到的自一致性与答案准确率之间的相关性，我们采用自一致性作为初始评估分数。随后，我们对生成相同答案的提示词的分数进行优化，使其保持一致。实验结果表明，即使在没有金标的情况下，GLaPE也能提供与准确率一致的可靠评估。此外，在六个流行的推理任务中，基于GLaPE的提示词优化所生成的有效提示词与基于准确率的提示词相当。代码公开于https://github.com/thunderous77/GLaPE。
英文摘要：
Despite the rapid progress of large language models (LLMs), their task performance remains sensitive to prompt design. Recent studies have explored leveraging the LLM itself as an optimizer to identify optimal prompts that maximize task accuracy. However, when evaluating prompts, such approaches heavily rely on elusive manually annotated gold labels to calculate task accuracy for each candidate prompt, which hinders the widespread implementation and generality. To overcome the limitation, this work proposes a gold label-agnostic prompt evaluation (GLaPE) to alleviate dependence on gold labels. Motivated by the observed correlation between self-consistency and the accuracy of the answer, we adopt self-consistency as the initial evaluation score. Subsequently, we refine the scores of prompts producing identical answers to be mutually consistent. Experimental results show that GLaPE provides reliable evaluations uniform with accuracy, even in the absence of gold labels. Moreover, on six popular reasoning tasks, our GLaPE-based prompt optimization yields effective prompts comparable to accuracy-based ones. The code is publicly available at https://github.com/thunderous77/GLaPE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
