Prompting Science Report 1: Prompt Engineering is Complicated and Contingent
Generative AI Labs, The Wharton School of Business, University of Pennsylvania(宾夕法尼亚大学沃顿商学院生成式人工智能实验室);WHU–Otto Beisheim School of Management(WHU-奥托·贝希姆管理学院);Glowforge(Glowforge公司)
Lennart Meincke, Ethan Mollick, Lilach Mollick, Dan Shapiro
本文是一系列简短报告中的第一篇,旨在帮助商业、教育和政策领域的领导者通过严谨的测试来理解使用人工智能的技术细节。在本报告中,我们展示了两点:\n- 对于衡量一个大语言模型(LLM)是否通过基准测试,不存在单一标准,而且选择何种标准会对该 LLM 在基准测试上的表现产生重大影响。你所选择的标准将取决于你在特定场景中使用 LLM 的目标。\n- 很难事先知道某种特定的提示方法会帮助还是损害 LLM 回答某个特定问题的能力。具体而言,我们发现有时对 LLM 保持礼貌有助于提升性能,有时则会降低性能。我们还发现,约束 AI 的回答在某些情况下有助于提升性能,但在其他情况下可能会降低性能。\n综合来看,这表明对 AI 性能进行基准测试并非“一刀切”,同时也表明特定的提示公式或方法(例如对 AI 保持礼貌)并非普遍有效。
英文摘要:
This is the first of a series of short reports that seek to help business, education, and policy leaders understand the technical details of working with AI through rigorous testing. In this report, we demonstrate two things:
- There is no single standard for measuring whether a Large Language Model (LLM) passes a benchmark, and that choosing a standard has a big impact on how well the LLM does on that benchmark. The standard you choose will depend on your goals for using an LLM in a particular case.
- It is hard to know in advance whether a particular prompting approach will help or harm the LLM's ability to answer any particular question. Specifically, we find that sometimes being polite to the LLM helps performance, and sometimes it lowers performance. We also find that constraining the AI's answers helps performance in some cases, though it may lower performance in other cases.
Taken together, this suggests that benchmarking AI performance is not one-size-fits-all, and also that particular prompting formulas or approaches, like being polite to the AI, are not universally valuable.
Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theory
MAIS, Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);University of California, Santa Barbara(加州大学圣巴巴拉分校);Beijing Wenge Technology Co., Ltd(北京文格科技有限公司)
Prompting Multi-Modal Tokens to Enhance End-to-End Autonomous Driving Imitation Learning with LLMs
University of Technology Sydney(悉尼科技大学);Australia Artificial Intelligence Institute(澳大利亚人工智能研究院);School of Computer Science(计算机学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
SIC-Agents: Benchmarking and Building an Adaptive Simulator for Pediatric Serious Illness Communication Training
McGill University(麦吉尔大学);Mila – Quebec AI Institute(米拉-魁北克人工智能研究所);Institute of Health Sciences Education, McGill University(麦吉尔大学健康科学教育研究所);School of Social Work, McGill University(麦吉尔大学社会工作学院);Montreal Children’s Hospital, McGill University Health Centre(麦吉尔大学健康中心蒙特利尔儿童医院);Steinberg Centre for Simulation and Interactive Learning, McGill University(麦吉尔大学斯坦伯格模拟与互动学习中心);Linarite AI(利纳里特人工智能公司)
SoM-1K: A Thousand-Problem Benchmark Dataset for Strength of Materials
College of Civil Engineering, Hunan University(湖南大学土木工程学院);Department of Civil & Architectural Engineering, University of Miami(迈阿密大学土木与建筑工程系);Department of Electrical and Computer Engineering, University of Miami(迈阿密大学电气与计算机工程系);School of Architecture, University of Miami(迈阿密大学建筑学院);Department of Computer Science, University of Illinois Chicago(伊利诺伊大学芝加哥分校计算机科学系)