C-MOP:结合动量和边界感知聚类以提升提示进化
C-MOP: Integrating Momentum and Boundary-Aware Clustering for Enhanced Prompt Evolution
AI总结:
C-MOP通过结合动量和边界感知聚类技术,提升提示优化效果,实现对大型语言模型的高效提示进化。
摘要:
自动提示优化是提升大型语言模型(LLM)性能的一个有前途的方向。然而,现有方法常受到噪声和冲突更新信号的困扰。在本研究中,我们提出了C-MOP(基于聚类的动量优化提示),通过边界感知对比采样(BACS)和动量引导语义聚类(MGSC)来稳定优化。具体来说,BACS利用批量信息挖掘三元特征——难负样本、锚点和边界对——以精确表征正负提示样本的典型表示和决策边界。为了解决语义冲突,MGSC引入了一种具有时间衰减的文本动量机制,从迭代中波动的梯度中提炼出持久的一致性。广泛的实验表明,C-MOP在PromptWizard和ProTeGi等SOTA基线方法上表现一致,平均提升幅度为1.58%和3.35%。值得注意的是,C-MOP使一个具有3B激活参数的一般LLM能够超越一个70B的领域专用密集LLM,突显了其在推动精确提示进化方面的有效性。代码可在https://github.com/huawei-noah/noah-research/tree/master/C-MOP上获取。
英文摘要:
Automatic prompt optimization is a promising direction to boost the performance of Large Language Models (LLMs). However, existing methods often suffer from noisy and conflicting update signals. In this research, we propose C-MOP (Cluster-based Momentum Optimized Prompting), a framework that stabilizes optimization via Boundary-Aware Contrastive Sampling (BACS) and Momentum-Guided Semantic Clustering (MGSC). Specifically, BACS utilizes batch-level information to mine tripartite features--Hard Negatives, Anchors, and Boundary Pairs--to precisely characterize the typical representation and decision boundaries of positive and negative prompt samples. To resolve semantic conflicts, MGSC introduces a textual momentum mechanism with temporal decay that distills persistent consensus from fluctuating gradients across iterations. Extensive experiments demonstrate that C-MOP consistently outperforms SOTA baselines like PromptWizard and ProTeGi, yielding average gains of 1.58% and 3.35%. Notably, C-MOP enables a general LLM with 3B activated parameters to surpass a 70B domain-specific dense LLM, highlighting its effectiveness in driving precise prompt evolution. The code is available at https://github.com/huawei-noah/noah-research/tree/master/C-MOP.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。