AI 中文总结
针对多租户LLM服务的缓存公平性问题，提出受生物物理启发的多类缓存替换策略，在合成工作负载上缩小了LRU与Belady的未命中率差距，实现公平性与吞吐量的可调权衡。
AI 中文摘要
多租户大语言模型（LLM）服务场景下的缓存替换是一个多类问题：短且高复用的系统提示词、长且中等复用的用户文档、中等长度的代码上下文以及突发的对话历史共享同一个驱逐池。在偏斜的多类到达情况下，传统的平面LRU策略仅将服务最差类的未命中率（$m_{\text{max}}$）暴露为一个不动点。我们引入一类由每类通量公式参数化的缓存替换策略，其中三个结构承诺——单个全局令牌质量不平衡信号$K$、并行的整流每类提升累加器，以及按年龄排序的驱逐后备——产生涌现的多类公平性。我们用线性V耦合整流通量和Goldman-Hodgkin-Katz（GHK）扩展实例化该类，其$V \to 0$的极限恰好是线性形式。在合成多类工作负载的四个偏斜级别上，该策略类将LRU到Belady的$m_{\text{max}}$差距缩小了27%至72%，线性和GHK在搜索方差内的核心目标上可互换。公平性/吞吐量权衡在单个超参数轴上表现为可调旋钮。我们将其与LeCaR反馈控制器谱系和形式控制理论的缓存衰减谱系相对比，作为已知成分的新颖组合。代码和复现脚本：this https URL
英文摘要
Cache replacement under multi-tenant LLM-serving conditions is a multi-class problem: short, high-reuse system prompts; long, moderate-reuse user documents; medium-length code context; and bursty conversation history share a single eviction pool. Under skewed multi-class arrivals, conventional flat-LRU policies expose the worst-served-class miss ratio ($m_{\max}$) only as a fixed point. We introduce a class of cache-replacement policies parameterised by a per-class flux formula, where three structural commitments -- a single global token-mass imbalance signal, $K$ parallel rectified per-class promotion accumulators, and an age-ordered eviction backstop -- produce emergent multi-class fairness. We instantiate this class with a linear V-coupled rectified flux and a Goldman-Hodgkin-Katz extension whose $V \to 0$ limit is exactly the linear form. Across four skew levels on synthetic multi-class workloads, the policy class closes 27--72\,\% of the LRU$\to$Belady gap on $m_{\max}$, with linear and GHK interchangeable on the headline objective within search variance. The fairness/throughput tradeoff is exposed as a tunable knob on a single hyperparameter axis. We position this against the LeCaR feedback-controller lineage and the formal-control-theory cache-decay lineage as a novel combination of known ingredients. Code and reproduction scripts: https://github.com/flatmax/membrane.cache
Comments24 pages, 1 figure
