The remarkable instruction-following capability of large language models (LLMs) has sparked a growing interest in automatically finding good prompts, i.e., prompt optimization. Most existing works follow the scheme of selecting from a pre-generated pool of candidate prompts. However, these designs mainly focus on the generation strategy, while limited attention has been paid to the selection method. Especially, the cost incurred during the selection (e.g., accessing LLM and evaluating the responses) is rarely explicitly considered. To overcome this limitation, this work provides a principled framework, TRIPLE, to efficiently perform prompt selection under an explicit budget constraint. TRIPLE is built on a novel connection established between prompt optimization and fixed-budget best arm identification (BAI-FB) in multi-armed bandits (MAB); thus, it is capable of leveraging the rich toolbox from BAI-FB systematically and also incorporating unique characteristics of prompt optimization. Extensive experiments on multiple well-adopted tasks using various LLMs demonstrate the remarkable performance improvement of TRIPLE over baselines while satisfying the limited budget constraints. As an extension, variants of TRIPLE are proposed to efficiently select examples for few-shot prompts, also achieving superior empirical performance.
From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions
University of Houston(休斯顿大学);University of California, Irvine(加州大学尔湾分校);Universidad Internacional de la Rioja(里瓦达维亚国际大学);University of Southern California(南加州大学);Abilene Christian University(阿比林基督教大学);University of Auckland(奥克兰大学);Aalto University(阿尔托大学);Carnegie Mellon University(卡内基梅隆大学);Indian Institute of Technology, Kanpur(印度理工学院,坎pur)
LIDS: LLM Summary Inference Under the Layered Lens
Department of Mathematics, University of Southern California(数学系,南加州大学);Data Sciences and Operations Department, Marshall School of Business, University of Southern California(数据科学与运营系,马歇尔商学院,南加州大学)
Combinatorial Multivariant Multi-Armed Bandits with Applications to Episodic Reinforcement Learning and Beyond
The Chinese University of Hong Kong(香港中文大学);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);Microsoft Research(微软研究院);Shanghai Jiao Tong University(上海交通大学)
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Institute for Clarity in Documentation(文档清晰性研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗克库尔中心);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室)
A Survey of Automatic Prompt Engineering: An Optimization Perspective
School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);East China Normal University(华东师范大学);Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(同济大学上海智能自主系统研究院)
CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding
Instituto Politécnico Nacional(墨西哥国立理工学院);Haverford College(哈弗福德学院);Wollo University(沃尔洛大学);Pennsylvania State University(宾夕法尼亚州立大学);University of Hamburg(汉堡大学)
Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
Hacettepe University(哈杰泰佩大学);Aselsan Research(阿塞尔桑研究院);Koç University(科克大学);Amsterdam UMC, University of Amsterdam(阿姆斯特丹大学医学中心,阿姆斯特丹大学);Amsterdam Public Health(阿姆斯特丹公共卫生研究院)
Compute Allocation for Self-Evolving LLMs: From Depth-Breadth to Multi-Armed Bandits
University of Notre Dame(诺丁汉大学);Northeastern University(东北大学);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);Southeast University(东南大学);Northwestern University(西北大学);Nanjing University(南京大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Asymptotically and Minimax Optimal Regret Bounds for Multi-Armed Bandits with Abstention
Institute of Operations Research and Analytics(运营研究与分析研究所);National University of Singapore(新加坡国立大学);Department of Electrical and Computer Engineering(电子与计算机工程系);Department of Mathematics(数学系)
Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting
University of Bayreuth(拜罗伊特大学)
我们提出了一个概念上的观点提示大语言模型(LLM),区分(1)diegetic提示(叙事的一部分,例如:“从前,我看到一只狐狸.),以及(2)非diegetic提示(外部,例如“写关于狐狸的冒险。").有了这个镜头,我们研究了129多产人群工作者如何用不同的用户界面(1 vs 3建议,有/没有非diegetic提示;实现GPT-3)写短文本:当界面提供多个建议,并提供了一个选项,非diegetic提示,参与者更喜欢从多个建议中选择,而不是通过非diegetic提示控制它们。当参与者提供非diegetic提示,这是要求灵感,主题或事实。特别是单一的建议,指导与diegetic和非diegetic信息。这项工作通过揭示(1)编写非diegetic提示需要努力,(2)人们结合diegetic和非diegetic提示,(3)他们使用他们的草稿(即diegetic信息)和建议时间来战略性地指导LLM,从而告知人类与人工智能的交互。
Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models
PythaLab, Yıldız Technical University, Istanbul, Turkey(PythaLab,Yıldız技术大学,伊斯坦布尔,土耳其)
Rethinking Prompt Optimizers: From Prompt Merits to Optimization
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore(南洋理工大学电子与电气工程学院);Home Team Science and Technology Agency, Singapore(新加坡科技局)
Meta-Prompt Optimization for LLM-Based Sequential Decision Making
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));The Chinese University of Hong Kong(香港中文大学);Guangdong Lab of AI and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳))
Large Language Models Have Intrinsic Meta-Cognition, but Need a Good Lens
School of Computer Science and Engineering, Key Laboratory of Computer Network and Information Integration, Ministry of Education, Southeast University, China(计算机科学与工程学院、计算机网络与信息集成重点实验室、教育部、东南大学,中国);Department of Neurosurgery, Shanghai Tenth People’s Hospital, School of Clinical Medicine of Nanjing Medical University, China(神经外科科、上海第十人民医院、南京医科大学临床医学学院,中国)