arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.04818cs.CLcs.AI
收藏

提示科学报告 1:提示工程是复杂且因情境而异的

Prompting Science Report 1: Prompt Engineering is Complicated and Contingent

Generative AI Labs, The Wharton School of Business, University of Pennsylvania(宾夕法尼亚大学沃顿商学院生成式人工智能实验室);WHU–Otto Beisheim School of Management(WHU-奥托·贝希姆管理学院);Glowforge(Glowforge公司)

Lennart Meincke, Ethan Mollick, Lilach Mollick, Dan Shapiro

AI总结:

本文通过实证测试指出,LLM 基准测试标准的选择会显著影响评估结果,且礼貌用语、约束回答等提示策略的效果因情境而异,并非普遍有效。

摘要:

本文是一系列简短报告中的第一篇,旨在帮助商业、教育和政策领域的领导者通过严谨的测试来理解使用人工智能的技术细节。在本报告中,我们展示了两点:\n- 对于衡量一个大语言模型(LLM)是否通过基准测试,不存在单一标准,而且选择何种标准会对该 LLM 在基准测试上的表现产生重大影响。你所选择的标准将取决于你在特定场景中使用 LLM 的目标。\n- 很难事先知道某种特定的提示方法会帮助还是损害 LLM 回答某个特定问题的能力。具体而言,我们发现有时对 LLM 保持礼貌有助于提升性能,有时则会降低性能。我们还发现,约束 AI 的回答在某些情况下有助于提升性能,但在其他情况下可能会降低性能。\n综合来看,这表明对 AI 性能进行基准测试并非“一刀切”,同时也表明特定的提示公式或方法(例如对 AI 保持礼貌)并非普遍有效。

英文摘要:

This is the first of a series of short reports that seek to help business, education, and policy leaders understand the technical details of working with AI through rigorous testing. In this report, we demonstrate two things: - There is no single standard for measuring whether a Large Language Model (LLM) passes a benchmark, and that choosing a standard has a big impact on how well the LLM does on that benchmark. The standard you choose will depend on your goals for using an LLM in a particular case. - It is hard to know in advance whether a particular prompting approach will help or harm the LLM's ability to answer any particular question. Specifically, we find that sometimes being polite to the LLM helps performance, and sometimes it lowers performance. We also find that constraining the AI's answers helps performance in some cases, though it may lower performance in other cases. Taken together, this suggests that benchmarking AI performance is not one-size-fits-all, and also that particular prompting formulas or approaches, like being polite to the AI, are not universally valuable.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑