arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2507.09019cs.LGcs.AIcs.DC
收藏

关于评估 LLM 推理服务系统性能的研究

On Evaluating Performance of LLM Inference Serving Systems

Amey Agrawal, Nitin Kedia, Anmol Agarwal, Jayashree Mohan, Nipun Kwatra, Souvik Kundu, Ramachandran Ramjee, Alexey Tumanov

AI总结:

本文指出了当前 LLM 推理系统评估中存在的反模式,提出了一套识别和避免这些反模式的综合检查清单与框架,并通过投机解码案例研究展示了其实际应用,为建立严谨的评估方法奠定了基础。

摘要:

大型语言模型(LLM)推理系统的快速演进带来了显著的效率提升。然而,我们的系统分析表明,当前的评估方法经常表现出根本性缺陷,通常表现为常见的评估反模式,这些反模式掩盖了真实的性能特征并阻碍了科学进步。通过对近期系统的全面检查,我们发现了在三个关键维度上反复出现的反模式:基线公平性、评估设置和指标设计。由于 LLM 推理具有结合了不同预填充和解码操作的双阶段性质,其对高度异构工作负载的处理,以及对交互式使用的严格时间要求,这些反模式对 LLM 推理来说尤为棘手。我们展示了常见的反模式——例如将工程投入与算法创新混为一谈的不充分基线比较、未能代表生产场景的工作负载选择,以及掩盖生成停滞等大量性能差异的指标归一化——是如何导致误导性结论的。为了应对这些挑战,我们根据分析提供了一份全面的检查清单,建立了一个用于识别和避免这些反模式的框架,以支持稳健的 LLM 推理评估。为了展示我们框架的实际应用,我们提出了一个分析投机解码的案例研究,该技术的突发性、非均匀 token 生成在使用这些反模式特征的方法进行评估时很容易被误解。我们的工作为评估方法奠定了严格的基础,实现了有意义的比较,确保了可重复的结果,并最终通过超越常见反模式使评估与现实世界需求相一致,加速了 LLM 推理系统的真正进步。

英文摘要:

The rapid evolution of Large Language Model (LLM) inference systems has yielded significant efficiency improvements. However, our systematic analysis reveals that current evaluation methodologies frequently exhibit fundamental flaws, often manifesting as common evaluation anti-patterns that obscure true performance characteristics and impede scientific progress. Through a comprehensive examination of recent systems, we identify recurring anti-patterns across three key dimensions: Baseline Fairness, Evaluation Setup, and Metric Design. These anti-patterns are uniquely problematic for LLM inference due to its dual-phase nature combining distinct prefill and decode operations, its handling of highly heterogeneous workloads, and its strict temporal requirements for interactive use. We demonstrate how common anti-patterns -- such as inadequate baseline comparisons that conflate engineering effort with algorithmic novelty, workload selections that fail to represent production scenarios, and metric normalizations that hide substantial performance variability like generation stalls-lead to misleading conclusions. To address these challenges, we provide a comprehensive checklist derived from our analysis, establishing a framework for recognizing and avoiding these anti-patterns in favor of robust LLM inference evaluation. To demonstrate the practical application of our framework, we present a case study analyzing speculative decoding, a technique whose bursty, non-uniform token generation is easily misinterpreted when evaluated using approaches characteristic of these anti-patterns. Our work establishes a rigorous foundation for evaluation methodology, enabling meaningful comparisons, ensuring reproducible results, and ultimately accelerating genuine progress in LLM inference systems by moving beyond common anti-patterns to align evaluation with real-world requirements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑