我们负担得起完美提示词吗？利用 Economical Prompting Index 平衡成本与准确率
Can We Afford The Perfect Prompt? Balancing Cost and Accuracy with the Economical Prompting Index
AI总结：
提出融合准确率、token 消耗与成本关注水平的 EPI 指标，跨模型和数据集评估提示技术，发现复杂方法未必更具成本效益。
摘要：
随着提示词工程研究迅速发展，准确率之外的评估对于开发具有成本效益的技术至关重要。我们提出 Economical Prompting Index（EPI），这是一种将准确率分数与 token 消耗量相结合的新型指标，并通过用户指定的成本关注水平进行调整，以反映不同的资源约束。我们的研究在 10 个广泛使用的语言模型和 4 个多样化数据集上检验了 6 种先进提示技术，包括 Chain-of-Thought、Self-Consistency 和 Tree of Thoughts。我们证明，Self-Consistency 等方法往往只带来统计上不显著的收益，同时成本却高得令人却步。例如，在 Claude 3.5 Sonnet 等高性能模型上，当成本关注水平较低时，Chain-of-Thought 等更简单技术的 EPI（0.72）超过 Self-Consistency 等更复杂方法（0.64）。我们的发现表明，应在资源受限场景中重新评估复杂提示策略，这可能重塑未来研究重点，并提升终端用户的成本效益。
英文摘要：
As prompt engineering research rapidly evolves, evaluations beyond accuracy are crucial for developing cost-effective techniques. We present the Economical Prompting Index (EPI), a novel metric that combines accuracy scores with token consumption, adjusted by a user-specified cost concern level to reflect different resource constraints. Our study examines 6 advanced prompting techniques, including Chain-of-Thought, Self-Consistency, and Tree of Thoughts, across 10 widely-used language models and 4 diverse datasets. We demonstrate that approaches such as Self-Consistency often provide statistically insignificant gains while becoming cost-prohibitive. For example, on high-performing models like Claude 3.5 Sonnet, the EPI of simpler techniques like Chain-of-Thought (0.72) surpasses more complex methods like Self-Consistency (0.64) at slight cost concern levels. Our findings suggest a reevaluation of complex prompting strategies in resource-constrained scenarios, potentially reshaping future research priorities and improving cost-effectiveness for end-users.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
