arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.08775cs.HCcs.AI
收藏

提示中应工程什么?基于需求驱动的LLM使用训练

What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use

Carnegie Mellon University(卡内基梅隆大学);Columbia University(哥伦比亚大学);University of Washington(华盛顿大学)

Qianou Ma, Weirui Peng, Chenyang Yang, Hua Shen, Kenneth Koedinger, Tongshuang Wu

AI总结:

本文提出ROPE框架,通过评估和训练套件帮助用户生成清晰需求,提升LLM应用构建效果。

摘要:

为复杂任务(如构建TripAdvisor聊天机器人)提示LLM需要人类明确表达定制化需求(如“以tl;dr开头”)。现有提示工程指令缺乏对需求表达的专注训练,而更强调可自动化的策略(如添加角色扮演和“逐步思考”技巧)。为弥补这一差距,我们引入需求导向的提示工程(ROPE),一种将人类注意力集中在生成清晰、完整需求的范式。我们通过评估和训练套件实现ROPE,提供有意识的练习与LLM生成反馈。在30名新手的随机对照实验中,ROPE显著优于传统提示工程训练(20% vs. 1%提升),这一差距自动提示优化无法弥补。此外,我们展示了输入需求质量与LLM输出之间的直接相关性。我们的工作为更多终端用户构建复杂LLM应用铺平了道路。

英文摘要:

Prompting LLMs for complex tasks (e.g., building a trip advisor chatbot) needs humans to clearly articulate customized requirements (e.g., "start the response with a tl;dr"). However, existing prompt engineering instructions often lack focused training on requirement articulation and instead tend to emphasize increasingly automatable strategies (e.g., tricks like adding role-plays and "think step-by-step"). To address the gap, we introduce Requirement-Oriented Prompt Engineering (ROPE), a paradigm that focuses human attention on generating clear, complete requirements during prompting. We implement ROPE through an assessment and training suite that provides deliberate practice with LLM-generated feedback. In a randomized controlled experiment with 30 novices, ROPE significantly outperforms conventional prompt engineering training (20% vs. 1% gains), a gap that automatic prompt optimization cannot close. Furthermore, we demonstrate a direct correlation between the quality of input requirements and LLM outputs. Our work paves the way to empower more end-users to build complex LLM applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑