Herd:通过智能组合器使用多个较小的 LLM 来匹配专有大型 LLM 的性能
Herd: Using multiple, smaller LLMs to match the performances of proprietary, large LLMs via an intelligent composer
AI总结:
本文提出 Herd 框架,通过智能路由器组合多个较小的开源 LLM,在匹配 ChatGPT 准确率的同时降低模型规模,并在 GPT 无法回答时以至少 40% 的概率找到可用模型。
摘要:
目前,存在超过一千个多用途且能够执行现实世界任务(包括问答、文本摘要、内容生成等)的 LLM。然而,免费模型的可用性、规模和可靠性阻碍了它们在日常用例中的广泛部署。为了解决访问和规模这两个问题,HuggingFace 等组织创建了模型仓库,用户在其中上传了模型权重、使用不同范式训练的模型的量化版本,以及描述其训练过程的模型卡片。虽然一些模型报告了在常用基准上的性能,但并非所有模型都如此,并且权衡基准性能与模型部署成本对现实世界影响的理解尚不明确。在这里,我们展示了一群开源模型可以通过智能路由器匹配或超越专有模型的性能。我们展示了由开源模型组成的 Herd 能够匹配 ChatGPT 的准确率,尽管它由实际上小 2.5 倍的模型组成。我们展示了在 GPT 无法回答查询的情况下,Herd 能够在至少 40% 的时间里识别出一个可以回答的模型。
英文摘要:
Currently, over a thousand LLMs exist that are multi-purpose and are capable of performing real world tasks, including Q&A, text summarization, content generation, etc. However, accessibility, scale and reliability of free models prevents them from being widely deployed in everyday use cases. To address the first two issues of access and scale, organisations such as HuggingFace have created model repositories where users have uploaded model weights and quantized versions of models trained using different paradigms, as well as model cards describing their training process. While some models report performance on commonly used benchmarks, not all do, and interpreting the real world impact of trading off performance on a benchmark for model deployment cost, is unclear. Here, we show that a herd of open source models can match or exceed the performance of proprietary models via an intelligent router. We show that a Herd of open source models is able to match the accuracy of ChatGPT, despite being composed of models that are effectively 2.5x smaller. We show that in cases where GPT is not able to answer the query, Herd is able to identify a model that can, at least 40% of the time.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。