优化LLM使用成本
Towards Optimizing the Costs of LLM Usage
AI总结：
本文提出通过预测LLM输出质量并优化选择，在控制成本的同时提升质量，实现成本降低40%-90%、质量提升4%-7%的方法。
摘要：
生成式AI，尤其是大语言模型（LLM），如今被广泛应用于各种文档处理任务，如问答和摘要生成。然而，不同的LLM在不同任务上具有不同的能力，同时也具有不同的成本、分词方式和延迟。事实上，企业在其各自的应用场景中，已经承担了运营或使用LLM的巨大成本。在这项工作中，我们提出通过估计LLM的输出质量（无需实际调用LLM）来优化其使用成本，然后求解一个优化程序来选择LLM，以在质量感知和延迟感知的方式下，要么将成本控制在预算之内，要么最小化成本。我们提出了一个模型来预测LLM在摘要等文档处理任务上的输出质量，随后采用一个LP舍入算法来优化LLM的选择。我们从理论和实证两方面研究了质量与成本权衡的优化问题。我们进一步提出一个句子简化模型，以受控方式减少令牌数量。此外，我们提出了几种确定性启发式方法，以质量感知的方式减少令牌，并研究了应用这些启发式方法以优化质量与成本权衡的相关优化问题。我们不仅在企业数据集上，还在我们标注的开源数据集上对我们的方法进行了广泛的实证验证，结果表明我们比最接近的基线方法表现好得多。我们的方法将成本降低了40%至90%，同时将质量提高了4%至7%。我们将向社区发布标注的开源数据集，以供进一步研究和探索。
英文摘要：
Generative AI and LLMs in particular are heavily used nowadays for various document processing tasks such as question answering and summarization. However, different LLMs come with different capabilities for different tasks as well as with different costs, tokenization, and latency. In fact, enterprises are already incurring huge costs of operating or using LLMs for their respective use cases. In this work, we propose optimizing the usage costs of LLMs by estimating their output quality (without actually invoking the LLMs), and then solving an optimization routine for the LLM selection to either keep costs under a budget, or minimize the costs, in a quality and latency aware manner. We propose a model to predict the output quality of LLMs on document processing tasks like summarization, followed by an LP rounding algorithm to optimize the selection of LLMs. We study optimization problems trading off the quality and costs, both theoretically and empirically. We further propose a sentence simplification model for reducing the number of tokens in a controlled manner. Additionally, we propose several deterministic heuristics for reducing tokens in a quality aware manner, and study the related optimization problem of applying the heuristics optimizing the quality and cost trade-off. We perform extensive empirical validation of our methods on not only enterprise datasets but also on open-source datasets, annotated by us, and show that we perform much better compared to closest baselines. Our methods reduce costs by 40%- 90% while improving quality by 4%-7%. We will release the annotated open source datasets to the community for further research and exploration.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
