面向边缘设备的令牌级路由推理系统
Token Level Routing Inference System for Edge Devices
AI总结：
本文提出一种令牌级路由推理系统，让边缘设备上的小模型选择性调用云端大模型生成关键令牌，在M1 MacBook上以0.5B模型实现CommonsenseQA 60%的性能提升，且云端上传令牌不足7%。
摘要：
大语言模型（LLM）推理的计算复杂度显著制约了其在边缘设备上的部署效率。相比之下，小语言模型具有更快的解码速度和更低的资源消耗，但往往面临响应质量下降和对幻觉的易感性增强的问题。为了解决这一权衡问题，协作解码应运而生，即由大模型协助生成关键令牌，已成为一种有前景的解决方案。该范式通过大模型的选择性干预实现高质量推理，同时保持小模型的速度和效率，从而充分利用两种模型类型的优势。在本工作中，我们提出了一种新颖的协作解码推理系统，该系统允许小模型在设备端执行推理，同时选择性地咨询云端大模型以生成关键令牌。值得注意的是，该系统在M1 MacBook上仅使用一个0.5B模型，就在CommonsenseQA上实现了60%的性能提升，且上传到云端大模型生成的令牌比例不足7%。
英文摘要：
The computational complexity of large language model (LLM) inference significantly constrains their deployment efficiency on edge devices. In contrast, small language models offer faster decoding and lower resource consumption but often suffer from degraded response quality and heightened susceptibility to hallucinations. To address this trade-off, collaborative decoding, in which a large model assists in generating critical tokens, has emerged as a promising solution. This paradigm leverages the strengths of both model types by enabling high-quality inference through selective intervention of the large model, while maintaining the speed and efficiency of the smaller model. In this work, we present a novel collaborative decoding inference system that allows small models to perform on-device inference while selectively consulting a cloud-based large model for critical token generation. Remarkably, the system achieves a 60% performance gain on CommonsenseQA using only a 0.5B model on an M1 MacBook, with under 7% of tokens generation uploaded to the large model in the cloud.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
