arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.07985cs.DCcs.AI
收藏

Dooly: 一种配置无关、冗余感知的LLM推理模拟器

Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation

University of Texas at Austin(德克萨斯大学奥斯汀分校)

Joon Ha Kim, Geon-Woo Kim, Anoop Rachakonda, Daehyeok Kim

AI总结:

本文提出Dooly,一种能够忽略配置差异并高效处理冗余的LLM推理模拟器,通过单次推理过程和智能标签传播减少冗余 profiling 耗时,提升模拟精度和效率。

摘要:

选择最优的LLM推理配置需要在硬件、服务引擎、注意力后端和模型架构之间进行评估,因为没有单一选择在所有工作负载中表现最佳。基于配置的模拟器是标准工具,但它们硬编码操作集到特定配置,并重新对每个操作进行重新配置,这使得探索变得成本高昂。这种成本源于对结构理解的缺失:每个操作的每个输入维度都由模型配置或 incoming 请求决定。许多模型配置值(例如头大小、层数)在不同模型中重复出现,因此相同操作在许多配置中运行;一次扫描请求依赖的维度即可服务所有。我们提出了Dooly,利用这种结构实现配置无关、冗余感知的配置。Dooly执行一次推理过程,通过污点传播标记每个输入维度的来源,并仅对不在其延迟数据库中的操作进行选择性配置;状态操作如注意力通过重用服务引擎自身的初始化代码进行隔离,从而消除手动仪器化。它基于数据库构建延迟回归模型,该模型成为现有模拟器的即插即用后端。在两个GPU平台、三个注意力后端和多样的模型架构上,Dooly在TTFT上达到5%的MAPE精度,在TPOT上达到8%的精度,同时将12个模型的profiling GPU小时减少了56.4%。我们已开源Dooly在https://github.com/dooly-project。

英文摘要:

Selecting the optimal LLM inference configuration requires evaluation across hardware, serving engines, attention backends, and model architectures, since no single choice performs best across all workloads. Profile-based simulators are the standard tool, yet they hardcode their operation set to a specific configuration and re-profile every operation from scratch, making exploration prohibitively expensive. This cost stems from a missing structural understanding: every input dimension of each operation is fixed by the model configuration or determined by the incoming request. Many model-configuration values (e.g., head size, layer count) recur across models, so the same operation runs in many configurations; a single sweep over the request-dependent dimensions can serve them all. We present Dooly, which exploits this structure to achieve configuration-agnostic, redundancy-aware profiling. Dooly performs a single inference pass, labels each input dimension with its origin via taint propagation, and selectively profiles only operations absent from its latency database; stateful operations such as attention are isolated by reusing the serving engine's own initialization code, eliminating manual instrumentation. From this database, it builds latency regression models that serve as a drop-in backend for existing simulators. Across two GPU platforms, three attention backends, and diverse model architectures, Dooly achieves simulation accuracy within 5% MAPE for TTFT and 8% for TPOT while reducing profiling GPU-hours across 12 models by 56.4% compared to the existing profiling approach. We have open-sourced Dooly at https://dooly-project.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑