发表机构
KAIST（韩国科学技术院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对统一内存端侧LLM，提出相位感知的CPU检索调度方法PhaseGate，通过区分预填充与解码阶段设置并发限制，在保持LLM延迟的同时将检索吞吐量提升至固定策略的2倍。
AI 中文摘要
端侧助手在统一内存系统上同时运行基于GPU的LLM推理和CPU检索。在饱和的本地检索工作负载下，四个并发检索工作线程在两个M4系统上将第95百分位（p95）解码延迟提高了60-61%，而预填充延迟仅上升5.7-6.9%。我们研究在受控LLM工作负载下，将LLM相位作为独立CPU检索的准入信号。PHASEGATE为预填充和解码分别校准并发限制，在我们的基础M4配置上选择四个和一个。在积压队列下，它实现了最佳测试可行固定策略总检索吞吐量的2.0倍，同时在所有七次保留运行中，两个p95 LLM延迟指标均保持在无检索基线的1.25倍以内。一个相位盲对照TimeGate使用相同的两个限制，基于校准得出的调度，但不观察LLM相位。它实现了相似的检索吞吐量，但在每次运行中都违反了输出令牌延迟限制。M2和M2 Pro Mac mini复现了策略排序，而输出长度扫描显示，随着解码占据每个请求的更多部分，优势会缩小。
英文摘要
On-device assistants run GPU-based LLM inference alongside CPU retrieval on unified-memory systems. Under a saturated local-retrieval workload, four concurrent retrieval workers raise 95th-percentile (p95) decode latency by 60-61% on two M4 systems, whereas prefill latency rises by only 5.7-6.9%. We study LLM phase as an admission signal for independent CPU retrieval under controlled LLM workloads. PHASEGATE calibrates separate concurrency limits for prefill and decode, selecting four and one on our base-M4 configuration. Under a backlogged queue, it achieves 2.0 times the aggregate retrieval throughput of the best tested feasible fixed policy, with both p95 LLM latency metrics within 1.25 times their no-retrieval baselines in all seven held-out runs. A phase-blind control, TimeGate, uses the same two limits on a calibration-derived schedule without observing LLM phase. It achieves similar retrieval throughput but violates the output-token latency limit in every run. M2 and M2 Pro Mac minis reproduce the policy ordering, while output-length sweeps show that the advantage narrows as decode occupies more of each request.
Comments11 pages, 3 figures. Accepted at the NeurIPS 2026 Workshop on On-Device Intelligence: Foundation Models under Real-World Constraints
