Interaction with Large Language Models (LLMs) is primarily carried out via prompting. A prompt is a natural language instruction designed to elicit certain behaviour or output from a model. In theory, natural language prompts enable non-experts to interact with and leverage LLMs. However, for complex tasks and tasks with specific requirements, prompt design is not trivial. Creating effective prompts requires skill and knowledge, as well as significant iteration in order to determine model behavior, and guide the model to accomplish a particular goal. We hypothesize that the way in which users iterate on their prompts can provide insight into how they think prompting and models work, as well as the kinds of support needed for more efficient prompt engineering. To better understand prompt engineering practices, we analyzed sessions of prompt editing behavior, categorizing the parts of prompts users iterated on and the types of changes they made. We discuss design implications and future directions based on these prompt engineering practices.
Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）；Google Cloud AI Research（谷歌云人工智能研究院）；Google DeepMind（谷歌DeepMind）；Salesforce Research（salesforce研究院）；Yale University（耶鲁大学）；Sea AI Lab（Sea人工智能实验室）；University of Waterloo（滑铁卢大学）
When Do Model Internals Help? Exploring the Role of Representation Engineering in LLM Safety
State Key Laboratory of Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）；School of Computer Science, Peking University（北京大学计算机科学学院）
Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language
University of Auckland（奥克兰大学）；Indian Institute of Science（印度科学学院）
GitHub Copilot是一个人工智能模型，用于从自然语言问题描述自动生成源代码。自2022年6月以来，Copilot已正式免费提供给所有学生作为Visual Studio Code等开发环境的插件。之前探索OpenAI Codex（Copilot的底层模型）的工作表明，它在典型的CS1问题上表现良好，因此人们担心它会对入门编程课程的教学产生影响。然而，关于Copilot不能很好地执行的问题类型，或者关于学生在解决错误时可能与Copilot进行的自然语言交互，我们知之甚少。我们通过评估Copilot在166个编程问题的公开数据集上的性能来探索这些问题。我们发现，它成功地解决了大约一半的这些问题的第一次尝试，它解决了60%的剩余问题，只使用自然语言的问题描述的变化。我们认为，这种类型的提示工程，我们相信将成为一个标准的人类和Copilot之间的互动时，它最初失败，是一个潜在的有用的学习活动，促进计算思维技能，并有可能改变代码编写技能的发展的性质。
