arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.14972cs.SEcs.AIcs.LG
收藏

缩小规模以实现规模化:生产环境中用开源SLM替代OpenAI大语言模型的成本效益分析

Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Production

Jaseci Labs(贾塞奇实验室);University of Michigan(密歇根大学)

Chandra Irugalbandara, Ashish Mahendra, Roland Daynauth, Tharuka Kasthuri Arachchige, Jayanaka Dantanarayana, Krisztian Flautner, Lingjia Tang, Yiping Kang, Jason Mars

AI总结:

针对企业依赖专有LLM服务的弊端,本文提出系统性评估方法与开源工具SLaM,测试显示9款开源SLM的29个变体在产品功能中效果可比GPT-4,性能一致性更优且成本降低5~29倍。

摘要:

许多企业使用大语言模型(large language models, LLMs)服务(例如OpenAI的GPT-4)来打造具备AI能力的产品体验。这种对专有服务的依赖虽具备易用性和缩短解决方案落地时间的优势,但在模型控制权、性能可靠性、正常运行时间可预测性以及成本方面存在弊端。与此同时,大量可商用的开源小语言模型(small language models, SLMs)不断涌现。然而,它们是否足以替代现有能力仍不明确,且目前尚无一套可全面评估这些SLM的系统性方法。本文提出了一套系统性评估方法,并针对真实产品功能场景下,用现代开源SLM替代专有LLM时的特性与权衡进行了刻画。我们设计了开源自动化分析工具SLaM,可对使用任意SLM的产品功能开展定量与定性测试。借助SLaM,我们对比了现代SLM与现有面向客户、基于OpenAI GPT-4 API实现的方案的质量与性能特征。在9款SLM及其29个变体的测试中,我们观察到SLM可提供具有竞争力的结果,性能一致性显著提升,且成本较GPT-4降低5倍至29倍。

英文摘要:

Many companies use large language models (LLMs) offered as a service, like OpenAI's GPT-4, to create AI-enabled product experiences. Along with the benefits of ease-of-use and shortened time-to-solution, this reliance on proprietary services has downsides in model control, performance reliability, uptime predictability, and cost. At the same time, a flurry of open-source small language models (SLMs) has been made available for commercial use. However, their readiness to replace existing capabilities remains unclear, and a systematic approach to holistically evaluate these SLMs is not readily available. This paper presents a systematic evaluation methodology and a characterization of modern open-source SLMs and their trade-offs when replacing proprietary LLMs for a real-world product feature. We have designed SLaM, an open-source automated analysis tool that enables the quantitative and qualitative testing of product features utilizing arbitrary SLMs. Using SLaM, we examine the quality and performance characteristics of modern SLMs relative to an existing customer-facing implementation using the OpenAI GPT-4 API. Across 9 SLMs and their 29 variants, we observe that SLMs provide competitive results, significant performance consistency improvements, and a cost reduction of 5x~29x when compared to GPT-4.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑