Continuous prompt search offers a computationally efficient alternative to conventional parameter tuning in natural language processing tasks. Nevertheless, its practical effectiveness can be significantly hindered by the black-box nature and the inherent high-dimensionality of the objective landscapes. Existing methods typically mitigate these challenges by restricting the search to a randomly projected low-dimensional subspace. However, the effectiveness and underlying motivation of the projection mechanism remain ambiguous. In this paper, we first empirically demonstrate that despite the prompt space possessing a low-dimensional structure, random projections fail to adequately capture this essential structure. Motivated by this finding, we propose a projection-free prompt search method based on evolutionary strategies. By directly optimizing in the full prompt space with an adaptation mechanism calibrated to the intrinsic dimension, our method achieves competitive search capabilities without additional computational overhead. Furthermore, to bridge the generalization gap in few-shot scenarios, we introduce a confidence-based regularization mechanism that systematically enhances the model's confidence in the target verbalizers. Experimental results on seven natural language understanding tasks from the GLUE benchmark demonstrate that our proposed approach significantly outperforms existing baselines.
Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performance
Indiana University Bloomington(印第安纳大学伯明顿分校);Center for Complex Networks and Systems(复杂网络与系统中心)
Ultra-Low-Dimensional Prompt Tuning via Random Projection
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta(阿尔伯塔大学计算机科学系与阿尔伯塔机器智能研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能讲席)
Enhance Multi-domain Sentiment Analysis of Review Texts through Prompting Strategies
Hong Kong Baptist University(香港浸会大学);BNU-UIC Institute of AI and Future Networks(北师大-联合国际学院人工智能与未来网络研究院);BNU-HKBU United International College(北师大-港浸大联合国际学院);Beijing Normal University(北京师范大学)
Identifying Effective Strategies in Prompt Evolution for Stress and Emotion Prediction
Kyung Hee University(庆熙大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);Graphen, Inc.(Graphen公司);Vanderbilt University(范德堡大学);Monroe Carell Jr. Children’s Hospital at Vanderbilt(范德堡大学门罗·卡雷尔 Jr.儿童医院)
Transferable Modeling Strategies for Low-Resource LLM Tasks: A Prompt and Alignment-Based Approach
Carnegie Mellon University(卡内基梅隆大学);Georgia Institute of Technology(佐治亚理工学院);San Francisco State University(旧金山州立大学);Northeastern University(东北大学);Rutgers University(罗格斯大学)
An automatically discovered chain-of-thought prompt generalizes to novel models and datasets
Medical University of Vienna(维也纳医科大学);Johannes Kepler University Linz(林茨约翰内斯·开普勒大学);KTH Royal Institute of Technology(KTH皇家理工学院);Humboldt University Berlin(柏林洪堡大学)
DP-ES: Differentially Private Evolution Strategies for Prompt Optimization
National University of Defense Technology(国防科技大学);CRRC Zhuzhou Electric Locomotive Research Institute Co., Ltd.(中车株洲电力机车研究所有限公司);China Academy of Railway Sciences(中国铁道科学研究院)
No One Fits All: From Fixed Prompting to Learned Routing in Multilingual LLMs
Department of Computer Science and Information Engineering, National Taiwan University, Taiwan(国立台湾大学计算机科学与资讯工程学系);Institute of Information Science, Academia Sinica, Taiwan(台湾“中央研究院”信息科学研究所);AI Research Center (AINTU), National Taiwan University, Taiwan(国立台湾大学人工智能研究中心)
HPSS: Heuristic Prompting Strategy Search for LLM Evaluators
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University(清华大学对话人工智能(CoAI)小组);University of Electronic Science and Technology of China(电子科技大学);Zhipu AI(智谱AI);The Knowledge Engineering Group (KEG), Tsinghua University(清华大学知识工程小组(KEG))