arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.04668cs.OScs.AIcs.PF
收藏

弹性组:与操作系统进程协同调度的硬屏障大语言模型推理组的逐令牌成员变更

Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes

Independent Researcher(独立研究员)

Daeyeon Son

AI总结:

研究设备上大语言模型解码中CPU - SIMD计算与操作系统对核心资源需求冲突问题,提出弹性组机制,通过ACK锁定期协议实现逐令牌核心成员变更,提升吞吐量并保障正确性。

摘要:

设备上的大语言模型解码是硬屏障CPU - SIMD计算,每令牌需核心毫秒级时间,与操作系统对核心的持续需求冲突。本文提出Anima OS的弹性组,推理组是可调度实体,核心成员可逐令牌变更。核心机制是ACK锁定期协议,在真实机器上验证了其在模型推理中的有效性及性能优势。

英文摘要:

On-device LLM decoding is a hard-barriered CPU-SIMD computation that wants every core for milliseconds per token, while the rest of the OS wants those same cores continuously. A barriered gang cannot simply be dropped into a preemptive scheduler: an unannounced departure deadlocks a barrier, and an unannounced arrival silently corrupts logits. I present the elastic gang of Anima OS, a bare-metal x86-64 Rust kernel in which the inference gang is a first-class schedulable entity whose core membership may change between any two tokens. The core mechanism is an ACK-latched epoch protocol that never waits on a named core: a seqlock-style generation-tagged latch composed with RCU/epoch-style membership consent, so each token's participant set is the intersection of the cores the gang requested and the cores that acked the current epoch. An un-acked core is outside this token and joins at most one token later. Displaced general processes migrate and keep running; cores return to them the moment a generation ends. On a real AMD Zen 5 machine (8C/16T), inference output is bit-exact under verified per-token membership change on both a 135M and a 7B model, the property that makes elasticity safe in a kernel whose safety gate reads logits. Against fair static core partitions, elastic membership Pareto-dominates: at intermediate inference duty cycles it delivers 1.75x (25%), 1.52x (50%), and 1.28x (75%) the general throughput of a static 8-core split at equal or better inference throughput, recovers all eight stranded cores when inference is idle, and converges to the split at saturation. Returning a lent core costs 0.22 us (p50); acquiring a busy, tenant-occupied core costs one scheduling quantum (~16 ms): a running tenant is never preempted mid-slice. Decode throughput saturates at gang width 8, so ceding cores past the knee is nearly free: elasticity auto-sizes the gang online.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑