更多的 LLM 调用就是你所需的全部吗？探讨复合推理系统的缩放定律
Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems
AI总结：
本文研究了复合推理系统的缩放特性，发现Vote和Filter-Vote系统的性能随LM调用次数呈非单调变化，并基于查询难度多样性提出分析性缩放模型，可准确预测并优化最优调用次数。
摘要：
近期在语言任务中取得的许多最先进成果，是通过执行多次语言模型（LM）调用并聚合其响应的复合系统来实现的。然而，对于 LM 调用次数（例如，要求 LM 对每个问题回答多次并进行多数投票）如何影响此类复合系统的性能，目前了解甚少。在本文中，我们启动了对复合推理系统缩放特性的研究。我们从理论和实证两方面分析了 LM 调用次数如何影响 Vote 和 Filter-Vote 这两种最简单的复合系统设计的性能，它们通过多数投票聚合 LM 响应，并选择性地应用 LM 过滤器。令人惊讶地发现，在多种语言任务中，Vote 和 Filter-Vote 的性能作为 LM 调用次数的函数，均呈现出先上升后下降的趋势。我们的理论结果表明，这种非单调性是由于任务内部查询难度的多样性所致：更多的 LM 调用在“简单”查询上带来更高性能，但在“困难”查询上导致更低性能，当任务同时包含这两类查询时，就会出现非单调行为。基于这一洞察，我们能够从少量样本中计算出使系统性能最大化的 LM 调用次数，并为这两种系统定义了分析性缩放模型。实验表明，我们的缩放模型能够准确预测 Vote 和 Filter-Vote 系统的性能，从而找到最优的 LM 调用次数。
英文摘要：
Many recent state-of-the-art results in language tasks were achieved using compound systems that perform multiple Language Model (LM) calls and aggregate their responses. However, there is little understanding of how the number of LM calls - e.g., when asking the LM to answer each question multiple times and taking a majority vote - affects such a compound system's performance. In this paper, we initiate the study of scaling properties of compound inference systems. We analyze, theoretically and empirically, how the number of LM calls affects the performance of Vote and Filter-Vote, two of the simplest compound system designs, which aggregate LM responses via majority voting, optionally applying LM filters. We find, surprisingly, that across multiple language tasks, the performance of both Vote and Filter-Vote can first increase but then decrease as a function of the number of LM calls. Our theoretical results suggest that this non-monotonicity is due to the diversity of query difficulties within a task: more LM calls lead to higher performance on "easy" queries, but lower performance on "hard" queries, and non-monotone behavior can emerge when a task contains both types of queries. This insight then allows us to compute, from a small number of samples, the number of LM calls that maximizes system performance, and define an analytical scaling model for both systems. Experiments show that our scaling model can accurately predict the performance of Vote and Filter-Vote systems and thus find the optimal number of LM calls to make.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
