arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.16867cs.LGcs.CL
收藏

边缘设备上的高效推理

Efficient Reasoning on the Edge

Qualcomm AI Research(高通人工智能研究)
Yelysei Bondarenko, Thomas Hehn, Rob Hesselink, Romain Lepert, Fabio Valerio Massoli, Evgeny Mironov, Leyla Mirvakhabova, Tribhuvanesh Orekondy, Spyridon Stasis… 展开作者

Yelysei Bondarenko, Thomas Hehn, Rob Hesselink, Romain Lepert, Fabio Valerio Massoli, Evgeny Mironov, Leyla Mirvakhabova, Tribhuvanesh Orekondy, Spyridon Stasis, Andrey Kuzmin, Anna Kuzina, Markus Nagel, Ankita Nayak, Corrado Rainone, Ork de Rooij, Paul N Whatmough, Arash Behboodi, Babak Ehteshami Bejnordi

AI总结:

提出结合LoRA适配器、监督微调、强化学习预算控制、并行测试时缩放、动态适配器切换和KV缓存共享的方法,在资源受限的边缘设备上实现高效准确的推理。

摘要:

具有思维链推理的大型语言模型在复杂问题解决任务中达到了最先进的性能,但其冗长的推理轨迹和大的上下文需求使其不适用于边缘部署。这些挑战包括高令牌生成成本、大的KV缓存占用,以及在将推理能力蒸馏到用于移动设备的较小模型时的低效性。现有方法通常依赖于将较大模型的推理轨迹蒸馏到较小模型中,这些轨迹冗长且风格冗余,不适合设备端推理。在这项工作中,我们提出了一种轻量级方法,通过使用LoRA适配器结合监督微调,在小型LLM中实现推理。我们进一步通过在这些适配器上进行强化学习引入预算控制,显著减少响应长度,同时保持最小的精度损失。为了解决内存受限的解码问题,我们利用并行测试时缩放,在轻微延迟增加的情况下提高精度。最后,我们提出了一种动态适配器切换机制,仅在需要时激活推理,以及在提示编码期间的KV缓存共享策略,减少设备端推理的首令牌时间。在Qwen2.5-7B上的实验表明,我们的方法在严格的资源约束下实现了高效、准确的推理,使LLM推理在移动场景中变得实用。展示我们的解决方案在移动设备上运行的视频可在我们的项目页面上找到。

英文摘要:

Large language models (LLMs) with chain-of-thought reasoning achieve state-of-the-art performance across complex problem-solving tasks, but their verbose reasoning traces and large context requirements make them impractical for edge deployment. These challenges include high token generation costs, large KV-cache footprints, and inefficiencies when distilling reasoning capabilities into smaller models for mobile devices. Existing approaches often rely on distilling reasoning traces from larger models into smaller models, which are verbose and stylistically redundant, undesirable for on-device inference. In this work, we propose a lightweight approach to enable reasoning in small LLMs using LoRA adapters combined with supervised fine-tuning. We further introduce budget forcing via reinforcement learning on these adapters, significantly reducing response length with minimal accuracy loss. To address memory-bound decoding, we exploit parallel test-time scaling, improving accuracy at minor latency increase. Finally, we present a dynamic adapter-switching mechanism that activates reasoning only when needed and a KV-cache sharing strategy during prompt encoding, reducing time-to-first-token for on-device inference. Experiments on Qwen2.5-7B demonstrate that our method achieves efficient, accurate reasoning under strict resource constraints, making LLM reasoning practical for mobile scenarios. Videos demonstrating our solution running on mobile devices are available on our project page.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑