动态混合精度路由用于高效多步LLM交互
Dynamic Mixed-Precision Routing for Efficient Multi-step LLM Interaction
AI总结：
本文提出动态混合精度路由框架，通过自适应选择高精度与低精度LLM，在多步决策中实现准确率与成本的平衡。
摘要：
大型语言模型（LLMs）通过测试时的多步交互和推理实现长视距决策任务的强性能。尽管从业者普遍认为更高的任务成功率需要使用更大更强的LLM模型，但多步交互使用大LLM会带来可抑制的推理成本。为了解决这个问题，我们探索在长视距决策过程中使用低精度量化LLMs。基于交互步骤中多样敏感性的观察，我们提出了动态混合精度路由（DMR）框架，该框架在每个决策步骤中自适应地在高精度和低精度LLMs之间进行选择。路由器通过一个两阶段流程进行训练，包括基于KL散度的监督学习，以识别对精度敏感的步骤，随后是组相对策略优化（GRPO），以进一步提高任务成功率。在ALFWorld和WebShop上的实验表明，我们的方法在单精度基线之上实现了强大的准确率-成本权衡。
英文摘要：
Large language models (LLMs) achieve strong performance in long-horizon agentic tasks through multi-step interaction and reasoning at test time. However, repeatedly invoking these models incurs substantial inference cost over an episode. Post-training quantization (PTQ) reduces this cost, but using a fixed low precision throughout an episode can sharply lower task success. We observe that interaction steps differ sharply in their sensitivity to quantization: given the same context, decoding a step at low precision turns a success into a failure at only about 1% of the steps, which we call critical steps. Based on this observation, we propose Dynamic Mixed-Precision Routing (DMR), a framework that adaptively selects between high-precision and low-precision versions of the same LLM at each decision step. Leveraging an entire-space factorization to jointly predict a step's divergence and its effect on the outcome, DMR learns to recognize these critical steps by training only a lightweight router on the model's hidden state. By reusing a shared high-precision KV cache and the hidden states, DMR avoids redundant context prefill and a separate text encoder, enabling precision switching with less than 1 ms of overhead per step and adding less than 0.11% to step latency for routing. This mixed-precision design also enables nested quantization, reducing the memory footprint by 39% on average compared with FP16. Across three long-horizon agentic benchmarks with nested quantization, DMR achieves an average 2.44x decode speedup over FP16 at comparable task success rates, without fine-tuning the underlying model. On ALFWorld with GPTQ, DMR achieves a 1.71x decode speedup with 86.9% success, compared with 87.2% for FP16.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
