arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.13835cs.LGcs.AIcs.CLcs.DB
收藏

SMART:在保证准确率的前提下自动缩小语言模型以降低处理费用

SMART: Automatically Scaling Down Language Models with Accuracy Guarantees for Reduced Processing Fees

Cornell University(康奈尔大学)

Saehan Jo, Immanuel Trummer

AI总结:

提出SMART框架,通过性能分析阶段评估多个LLM并策略性组合不同模型,在保证用户设定准确率约束下显著降低NLP任务的推理成本。

摘要:

大型语言模型(LLMs)的进步显著提升了自然语言处理(NLP)任务的性能。然而,部署高性能的LLMs会产生高昂的成本,这主要是由于旨在增强模型性能的参数数量增加所致。这使得最终用户使用最先进的LLMs变得更加昂贵。诸如OpenAI和Anthropic等AI服务提供商通常提供多个版本的LLMs,这些版本具有不同的价格和性能。然而,最终用户在选择适合其任务的LLMs时仍面临挑战,难以在结果质量和成本之间取得平衡。我们引入了SMART(Scaling Models Adaptively for Reduced Token Fees,自适应缩放模型以减少Token费用),这是一种新颖的LLM框架,旨在最小化NLP任务的推理成本,同时确保足够的结果质量。它允许用户以输出与最强大LLM的输出等价的形式指定准确率约束。随后,SMART生成的结果偏离该LLM输出的概率低于用户定义的阈值。SMART采用性能分析阶段,评估多个LLMs的性能,以识别满足用户定义准确率水平的模型。SMART优化了性能分析开销与性能分析带来的预期成本节约之间的权衡。此外,我们的方法通过战略性地利用LLMs的组合显著降低了推理成本。我们在三个真实世界数据集上的实验表明,基于OpenAI模型,SMART实现了显著的成本节约,与GPT-4相比最高可达25.6倍。

英文摘要:

The advancement of Large Language Models (LLMs) has significantly boosted performance in natural language processing (NLP) tasks. However, the deployment of high-performance LLMs incurs substantial costs, primarily due to the increased number of parameters aimed at enhancing model performance. This has made the use of state-of-the-art LLMs more expensive for end-users. AI service providers, such as OpenAI and Anthropic, often offer multiple versions of LLMs with varying prices and performance. However, end-users still face challenges in choosing the appropriate LLM for their tasks that balance result quality with cost. We introduce SMART, Scaling Models Adaptively for Reduced Token Fees, a novel LLM framework designed to minimize the inference costs of NLP tasks while ensuring sufficient result quality. It enables users to specify an accuracy constraint in terms of the equivalence of outputs to those of the most powerful LLM. SMART then generates results that deviate from the outputs of this LLM only with a probability below a user-defined threshold. SMART employs a profiling phase that evaluates the performance of multiple LLMs to identify those that meet the user-defined accuracy level. SMART optimizes the tradeoff between profiling overheads and the anticipated cost savings resulting from profiling. Moreover, our approach significantly reduces inference costs by strategically leveraging a mix of LLMs. Our experiments on three real-world datasets show that, based on OpenAI models, SMART achieves significant cost savings, up to 25.6x in comparison to GPT-4.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑