发表机构
Shenzhen Technology University; The Hong Kong Polytechnic University; Zhejiang Lab; The Chinese University of Hong Kong; HKUST (Guangzhou)（深圳技术大学; 香港理工大学; 之江实验室; 香港中文大学; 香港科技大学（广州））
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对大型语言模型的提示词词汇敏感性开展大规模机制分析，揭示提示词性能稳定性缩放定律，提出自动化提示词优化智能体，可降低代码生成任务性能方差40.7%，为鲁棒提示工程提供可解释框架。
AI 中文摘要
大型语言模型（LLMs）对表层提示词变体表现出极强的敏感性，微小的词汇变化可引发不成比例的性能波动。我们跳出黑盒优化和粗粒度模板，利用含132000个提示词变体的数据集，开展首个大规模n元语法词元级的提示词稳定性机制分析。研究揭示了提示词性能稳定性的基础缩放定律：更高的平均任务性能与更低的方差、更强的提示词扰动鲁棒性密切相关。我们识别出支撑该鲁棒性的两大核心语言驱动因素：（1）领域特定术语，其紧密锚定语义边界；（2）显式动作指令，其形式化推理轨迹。这些要素共同约束模型的解释空间，有效“锁定”更具确定性的生成行为。基于此，我们引入自动化提示词优化智能体（Prompt-Refining Agent），通过注入领域锚定和操作约束系统重构输入查询。实证评估显示，该方法在代码生成任务中将性能方差降低40.7%，同时保持或提升平均性能。这些发现为实现鲁棒提示工程提供了基于统计且具机制可解释性的框架。
英文摘要
Large Language Models (LLMs) exhibit extreme sensitivity to surface-level prompt variations, in which minor lexical changes can trigger disproportionate performance fluctuations. Moving beyond black-box optimization and coarse-grained templates, we present the first large-scale, n-gram token-level mechanistic analysis of prompt stability, leveraging a dataset of 132,000 prompt variants. Our investigation reveals a fundamental Scaling Law of Prompt Performance Stability: higher average task performance is strongly associated with lower variance and greater robustness across prompt perturbation. We identify two core linguistic drivers underlying this robustness: (1) Domain-Specific Terminology, which tightly anchors semantic boundaries, and (2) Explicit Action Directives, which formalize reasoning trajectories. Together, these elements constrain the model's interpretative space, effectively ``locking in'' more deterministic generation behavior. Building on these insights, we introduce an automated Prompt-Refining Agent that systematically restructures input queries by injecting domain anchoring and operational constraints. Empirical evaluation shows that our approach reduces performance variance by 40.7% in code generation task, while preserving or improving mean performance. These findings provide a statistically grounded and mechanistically interpretable framework for achieving robust prompt engineering.
