arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.02640cs.SDcs.AIcs.NI
收藏

节拍器:限制缓存,为实时交互模型服务保持节奏

Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving

Pine AI(松果人工智能公司)

Jiaying Meng, Bojie Li

AI总结:

研究实时交互模型服务问题,核心方法是限制会话驻留状态。主要贡献是消除崩溃,使每帧延迟成为单调负载信号,在线准入控制器可发现可调度并发,还能用一阶模型预测崩溃时间。

摘要:

实时交互模型将服务变成周期性实时任务,存在危险故障模式。持续负载会使服务崩溃且难以察觉。本文表明限制每个会话的驻留状态可恢复稳定性和可观测性。Metronome的引擎内KV窗口消除崩溃,使延迟成为单调负载信号,一阶模型可预测崩溃时间,质量探针验证边界设计。

英文摘要:

Real-time interaction models -- Moshi, MiniCPM-o, Qwen-Omni -- turn serving into a periodic real-time task: on every frame a session ingests streaming audio and must respond by a recurring wall-clock deadline, while its KV cache grows monotonically and stays pinned for the whole conversation. This regime hides a dangerous failure mode. On a real full-duplex stack, sustained load does not degrade serving gracefully: it falls off a cliff, jumping in one step from milliseconds per frame to a stalled engine when accumulated session state exhausts the KV pool. The collapse is metastable -- identical five-minute runs collapse or survive on run-to-run variance -- and silent: latency and deadline-miss metrics read healthy throughout. We show one move restores both stability and observability: bound each session's resident state, and latency starts telling the truth. Metronome's in-engine KV window eliminates the collapse (0/20 vs. 14/20 runs across two batches) and turns per-frame latency into a monotone load signal, on which an online admission controller discovers the schedulable concurrency; without the window, the identical controller over-admits into the wall. A first-order model predicts the collapse time within a few percent on the headline model, and a quality probe validates the bound's design by ablation: the window alone is quality-free in turn-based decoding, and its few pinned attention-sink tokens are what keep free-running generation healthy. Everything is measured end-to-end on real audio, across four interaction models on one GPU.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑