arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.13786cs.CLcs.NE
收藏

基于进化策略的无投影连续提示搜索

Projection-Free Evolution Strategies for Continuous Prompt Search

Yu Cai, Canxi Huang, Xiaoyu He

AI总结:

本文提出无投影连续提示搜索方法,通过在完整提示空间中直接优化,提升搜索效率并减少计算开销,实验表明在GLUE基准的七项任务中优于现有基线。

摘要:

连续提示搜索为自然语言处理任务提供了一种计算高效的替代方案,但其实际效果常受限于黑盒性质和高维目标景观。现有方法通过限制搜索到随机投影的低维子空间来缓解这些问题,但投影机制的有效性和动机仍不明确。本文通过实验证明,尽管提示空间具有低维结构,随机投影无法充分捕捉该结构。受此启发,本文提出基于进化策略的无投影提示搜索方法,通过在完整提示空间中直接优化并校准适应机制以适应内在维度,实现无需额外计算开销的竞争性搜索能力。此外,为弥合少样本场景中的泛化差距,本文引入基于置信度的正则化机制,系统增强模型对目标词向量的置信度。实验结果表明,在GLUE基准的七项自然语言理解任务中,本文方法显著优于现有基线。

英文摘要:

Continuous prompt search offers a computationally efficient alternative to conventional parameter tuning in natural language processing tasks. Nevertheless, its practical effectiveness can be significantly hindered by the black-box nature and the inherent high-dimensionality of the objective landscapes. Existing methods typically mitigate these challenges by restricting the search to a randomly projected low-dimensional subspace. However, the effectiveness and underlying motivation of the projection mechanism remain ambiguous. In this paper, we first empirically demonstrate that despite the prompt space possessing a low-dimensional structure, random projections fail to adequately capture this essential structure. Motivated by this finding, we propose a projection-free prompt search method based on evolutionary strategies. By directly optimizing in the full prompt space with an adaptation mechanism calibrated to the intrinsic dimension, our method achieves competitive search capabilities without additional computational overhead. Furthermore, to bridge the generalization gap in few-shot scenarios, we introduce a confidence-based regularization mechanism that systematically enhances the model's confidence in the target verbalizers. Experimental results on seven natural language understanding tasks from the GLUE benchmark demonstrate that our proposed approach significantly outperforms existing baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑