LLM-Emu: 通过基于性能的采样实现LLM推理的原生运行时仿真
LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling
AI总结:
LLM-Emu通过基于性能的采样实现LLM推理的原生运行时仿真,能够准确模拟vLLM服务行为,支持在线实验。
摘要:
真实评估LLM服务系统需要在线工作负载、动态到达、排队和服务引擎的本地调度进行执行批处理,但在此类实验上GPU运行成本高。现有模拟器虽然降低了成本,但通常离线运行或时间拉伸模式,重新实现服务引擎调度器,或需要精确的操作员/内核级延迟模型。我们提出了LLM-Emu,这是一个为vLLM设计的仿真器,保留生产HTTP、调度、KV缓存和输出处理路径,仅将GPU前向执行替换为基于性能采样的延迟和合成输出令牌。在两种不同的GPU、四种模型变体、两种模型家族、两种注意力后端以及Poisson和突发性ShareGPT工作负载上测试,LLM-Emu紧密跟踪真实的vLLM服务行为:TPOT和ITL绝对误差在4.8%以内,E2E延迟在5.3%以内,输出吞吐量在1.9%以内;TTFT稳定性较差,最大误差为10.4%,反映其对准入和队列状态的敏感性。这些结果表明,轻量级的原生仿真可以支持LLM服务系统的实际在线实验。LLM-Emu在https://github.com/AKafakA/llm-emu开源。
英文摘要:
Realistic evaluation of LLM serving systems requires online workloads, dynamic arrivals, queueing, and the serving engine's local scheduling for execution batching, but running such experiments on GPUs is expensive. Existing simulators reduce this cost, but often operate offline or in time-warped mode, re-implement serving-engine schedulers, or require accurate operator/kernel-level latency models. We present LLM-Emu, a serving-native emulator for vLLM that preserves the production HTTP, scheduling, KV-cache, and output-processing paths while replacing only GPU forward execution with profile-sampled latency and synthetic output tokens. Tested on two different GPUs, four model variants, two model families, two attention backends, and both Poisson and bursty ShareGPT workloads, LLM-Emu closely tracks real vLLM serving behavior: TPOT and ITL stay within $4.8\%$ absolute error, E2E latency within $5.3\%$, and output throughput within $1.9\%$; TTFT is less stable, with maximum error $10.4\%$, reflecting its sensitivity to admission and queue state. These results suggest that lightweight, serving-native emulation can support practical online experimentation for LLM-serving systems. LLM-Emu is open sourced at https://github.com/AKafakA/llm-emu.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。