Power has become a central bottleneck for AI inference. This problem is becoming more urgent as agentic AI emerges as a major workload class, yet prior power-management techniques focus almost entirely on single-turn LLM serving. Our analysis shows that agentic serving behaves fundamentally differently: each request carries long-lived context that evolves across tool-interleaved turns, and lowering GPU frequency can push the system into a thrashing regime where memory pressure sharply worsens both performance and power efficiency. These observations show that power optimization for agentic serving requires rethinking.
We present KAIROS, a context-aware power optimization system for agentic AI serving. KAIROS uses agent context as a first-class control signal to jointly manage GPU frequency, per-instance concurrency, and multi-instance request placement. This enables KAIROS to save power when memory headroom exists while avoiding thrashing and preserving performance targets. At a high level, KAIROS tracks requests at agent granularity, adapts local control to context growth and agent progress, and routes agents across instances to jointly improve power efficiency and memory stability. Evaluated across diverse software and data engineering agentic tasks, KAIROS achieves an average of 27% (up to 39.8%) power reduction while meeting the performance targets.
[AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflows
University of Virginia, Biocomplexity Institute and Initiative(弗吉尼亚大学生物复杂性研究所与倡议);Rutgers University(罗格斯大学);Princeton Plasma Physics Laboratory(普林斯顿等离子体物理实验室)
Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework
NTU Singapore(南洋理工大学);AWS(亚马逊网络服务);Microsoft(微软);AMD(超威半导体);Ant Group(蚂蚁集团);NCSpeech(新加坡国立大学语音实验室);Imperial College London(帝国理工学院);University of Edinburgh(爱丁堡大学)
Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoC
School of Computer Science, Peking University(北京大学计算机科学学院);The University of Hong Kong(香港大学);National Key Laboratory for Multimedia Information Processing, Peking University(北京大学多媒体信息处理国家重点实验室)