重新思考提示策略在LLM测试时缩放中的作用：概率论视角
Rethinking the Role of Prompting Strategies in LLM Test-Time Scaling: A Perspective of Probability Theory
AI总结：
本文针对LLM测试时计算缩放场景，通过系统实验发现复杂提示策略随采样次数增加性能逐渐落后于简单思维链，提出概率预测方法及性能提升策略，为重新审视提示策略作用提供新视角。
摘要：
近年来，大语言模型（LLM）的测试时计算缩放受到广泛关注，但关于不同推理提示策略在缩放时的表现研究有限。本文聚焦于一种标准且现实的缩放设置：多数投票。我们系统地进行了6个LLM×8种提示策略×6个基准的实验。实验结果一致表明，随着采样次数和计算开销增加，初始性能更优的复杂提示策略逐渐落后于简单的思维链（Chain-of-Thought）。我们分析了这一现象并提供理论证明。此外，我们提出一种概率方法，可高效预测缩放性能并确定大采样次数下的最佳提示策略，无需实际应用中资源密集的推理过程。我们还基于理论分析引入两种显著提升缩放性能的方法。希望我们的研究能推动重新审视复杂提示的作用，释放简单提示策略的潜力，并为增强测试时缩放性能提供新见解。代码可在https://github.com/MraDonkey/rethinking_prompting获取。
英文摘要：
Recently, scaling test-time compute on Large Language Models (LLM) has garnered wide attention. However, there has been limited investigation of how various reasoning prompting strategies perform as scaling. In this paper, we focus on a standard and realistic scaling setting: majority voting. We systematically conduct experiments on 6 LLMs $\times$ 8 prompting strategies $\times$ 6 benchmarks. Experiment results consistently show that as the sampling time and computational overhead increase, complicated prompting strategies with superior initial performance gradually fall behind simple Chain-of-Thought. We analyze this phenomenon and provide theoretical proofs. Additionally, we propose a probabilistic method to efficiently predict scaling performance and identify the best prompting strategy under large sampling times, eliminating the need for resource-intensive inference processes in practical applications. Furthermore, we introduce two ways derived from our theoretical analysis to significantly improve the scaling performance. We hope that our research can promote to re-examine the role of complicated prompting, unleash the potential of simple prompting strategies, and provide new insights for enhancing test-time scaling performance. Code is available at https://github.com/MraDonkey/rethinking_prompting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
