Automatic prompt optimization reduces manual prompt engineering, but relies on task performance measured on a small, often randomly sampled evaluation subset as its main source of feedback signal. Despite this, how to select that evaluation subset is usually treated as an implementation detail. We study evaluation subset selection for prompt optimization from a principled perspective and propose SESS, a submodular evaluation subset selection method. We frame selection as maximizing an objective set function and show that, under mild conditions, it is monotone and submodular, enabling greedy selection with theoretical guarantees. Across GSM8K, MATH, and GPQA-Diamond, submodularly selected evaluation subsets can yield better optimized prompts than random or heuristic baselines.
Sub-SA: Strengthen In-context Learning via Submodular Selective Annotation
School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）；School of Automation, Southeast University（东南大学自动化学院）；State Key Laboratory of ASIC and System, Fudan University（复旦大学专用集成电路与系统国家重点实验室）
New York University（纽约大学）；University of California, San Diego（加州大学圣地亚哥分校）
Prompt-based learning (i.e., prompting) is an emerging paradigm for exploiting knowledge learned by a pretrained language model. In this paper, we propose Automatic Multi-Label Prompting (AMuLaP), a simple yet effective method to automatically select label mappings for few-shot text classification with prompting. Our method exploits one-to-many label mappings and a statistics-based algorithm to select label mappings given a prompt template. Our experiments demonstrate that AMuLaP achieves competitive performance on the GLUE benchmark without human effort or external resources.
Shanghai Jiao Tong University（上海交通大学）；Amazon Web Services（亚马逊云科技）
Large language models (LLMs) can perform complex reasoning by generating intermediate reasoning steps. Providing these steps for prompting demonstrations is called chain-of-thought (CoT) prompting. CoT prompting has two major paradigms. One leverages a simple prompt like "Let's think step by step" to facilitate step-by-step thinking before answering a question. The other uses a few manual demonstrations one by one, each composed of a question and a reasoning chain that leads to an answer. The superior performance of the second paradigm hinges on the hand-crafting of task-specific demonstration…
Experience-Guided Reflective Co-Evolution of Prompts and Heuristics for Automatic Algorithm Design
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学北京校区人工智能学院）；Department of Data Science, City University of Hong Kong（香港城市大学数据科学系）；School of Information, Renmin University of China（中国人民大学信息学院）；WeChat, Tencent（微信，腾讯）
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Tencent AI Lab（腾讯AI实验室）；Institute of Modern Languages and Linguistics, Fudan University（复旦大学现代语言与语言学研究所）
An automatically discovered chain-of-thought prompt generalizes to novel models and datasets
Medical University of Vienna（维也纳医科大学）；Johannes Kepler University Linz（林茨约翰内斯·开普勒大学）；KTH Royal Institute of Technology（KTH皇家理工学院）；Humboldt University Berlin（柏林洪堡大学）
Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；University of Liverpool（利物浦大学）
Discrete prompts have been used for fine-tuning Pre-trained Language Models for diverse NLP tasks. In particular, automatic methods that generate discrete prompts from a small set of training instances have reported superior performance. However, a closer look at the learnt prompts reveals that they contain noisy and counter-intuitive lexical constructs that would not be encountered in manually-written prompts. This raises an important yet understudied question regarding the robustness of automatically learnt discrete prompts when used in downstream tasks. To address this question, we conduct …
Institute of Software, CAS（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Microsoft（微软公司）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）
University of Toronto（多伦多大学）；Vector Institute（矢量研究所）；University of Waterloo（滑铁卢大学）
By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purpose computers. However, task performance depends significantly on the quality of the prompt used to steer the model, and most effective prompts have been handcrafted by humans. Inspired by classical program synthesis and the human approach to prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic instruction generation and selection. In our method, we treat the instruction as the "program," optimized by searching over a pool of instruction c…
Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring with Feedback
Graduate School of Information Science and Electrical Engineering（信息科学与电气工程研究生院）；Kyushu University（九州大学）；Faculty of Information Science and Electrical Engineering（信息科学与电气工程学部）
Prompt-R1: Collaborative Automatic Prompting Framework via End-to-end Reinforcement Learning
Hainan University（海南大学）；Nanyang Technological University（南洋理工大学）；Hithink Research（慧思研究）；HKUST (Guangzhou)（香港科技大学（广州））；IDEA Research（IDEA研究院）；Tsinghua University（清华大学）；National University of Singapore（新加坡国立大学）；Nanjing University of Science and Technology（南京理工大学）
