上下文并非免费:面向跨地理分布式数据中心的多轮LLM Web服务的碳感知LLM路由
Context Is Not Free: Carbon-Aware LLM Routing for Multi-Turn LLM Web Services across Geo-Distributed Data Centers
AI总结:
针对跨地理分布式数据中心的多轮LLM Web服务,提出CarbonRouter框架,通过缓存感知碳成本模型和轻量级门控机制,联合选择模型、GPU类型和区域,较FairCarbon平均减碳16.1%,长会话减碳达23.1%。
摘要:
Web的可持续性日益受到关注,因为基于大语言模型(LLM)的Web服务会产生大量碳足迹。这些服务通常运行在地理分布式数据中心中,其电网碳强度随区域和时间变化,因此将每个查询路由到更清洁的区域或更小的模型可减少排放。然而,对话是有状态的Web会话:每一轮都会复用历史的键值(KV)缓存,该缓存仅驻留在之前轮次的服务位置。在另一个部署上服务某一轮需要重新计算该历史,其碳成本随对话轮次增长,可能超过迁移带来的节省。现有的碳感知LLM路由器将对话的查询视为独立的,忽略了这一成本。我们提出CarbonRouter,这是一个在线框架,可联合为每个查询选择模型、GPU类型和区域以最小化碳排放。CarbonRouter引入了缓存感知的碳成本模型,该模型使用来自服务后端的缓存令牌计数,将历史重新计算成本计入每个候选部署,并使用轻量级门控机制,当会话可能在下一轮迁移并再次重建其历史时,将其保留在缓存部署上。我们在两个真实对话数据集上进行了跟踪驱动实验,使用五个LLM、三种GPU类型和四个区域,结果显示CarbonRouter在所有策略中实现了最低的碳排放量。与具有相同动作空间的最先进碳感知路由器FairCarbon相比,CarbonRouter平均减少16.1%的CO₂,同时还提高了预测答案质量和估计延迟。此外,对于超过30轮的会话,节省量进一步上升至23.1%。
英文摘要:
The sustainability of the Web has drawn growing attention, as large language model (LLM)-based web services leave a substantial carbon footprint. These services typically run in geo-distributed data centers whose grid carbon intensity varies across regions and over time, so routing each query to a cleaner region or a smaller model can reduce emissions. A conversation, however, is a stateful web session: each turn reuses the key--value (KV) cache of its history, which resides only where earlier turns were served. Serving a turn on another deployment requires recomputing this history, whose carbon cost grows with the conversation and can outweigh the saving of the move. Existing carbon-aware LLM routers treat a conversation's queries as independent and ignore this cost. We present CarbonRouter, an online framework that jointly selects the model, GPU type, and region for each query to minimize carbon emissions. CarbonRouter introduces a cache-aware carbon cost model, which charges history recomputation to every candidate deployment using cached-token counts from the serving backends, and uses a lightweight gate that keeps a session on its cached deployment when it would likely move back at the next turn and rebuild its history again. We evaluate CarbonRouter in trace-driven experiments on two real-world conversation datasets with five LLMs, three GPU types, and four regions, where it achieves the lowest carbon of all policies. CarbonRouter reduces CO$_2$ by 16.1\% on average relative to FairCarbon, a state-of-the-art carbon-aware router with the same action space, while also improving predicted answer quality and estimated latency. Moreover, the saving rises further to 23.1\% for sessions longer than 30 turns.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。