通过本地-云模型级联平衡代码补全的延迟与准确性
Balancing Latency and Accuracy of Code Completion via Local-Cloud Model Cascading
AI总结：
MCCom通过本地-云模型级联平衡代码补全的延迟与准确性，提升性能并降低成本
摘要：
行级代码补全需要在高准确性和低延迟之间取得关键平衡。现有方法面临取舍：大型语言模型（LLMs）提供高质量的建议但会产生高延迟，而小型语言模型（SLMs）速度较快但往往效果不佳。我们提出了MCCom（基于模型级联的代码补全）框架，该框架级联了一个本地SLM和基于云的LLM。为了实现有效的级联，MCCom利用用户行为作为新的信号来触发LLM，仅在SLM失败时才调用LLM，显著降低了云计算成本。此外，我们引入了两阶段推测解码策略和迭代检索机制以增强模型间的协作。我们还训练了一个12100万参数的轻量模型，其性能达到7B状态最先进模型的73.8%。在RepoEval和新的现实世界基准StmtEval上评估，MCCom将推理延迟降低了最高47.9%，减少了46.3%的LLM使用，同时通过有效的协作将LLM的精确匹配率提高了8.9%。
英文摘要：
Line-level code completion requires a critical balance between high accuracy and low latency. Existing methods suffer from a trade-off: large language models (LLMs) provide high-quality suggestions but incur high latency, while small language models (SLMs) are fast but often suboptimal. We propose MCCom (Model-Cascading-based code Completion), a framework that cascades a local SLM with a cloud-based LLM. To achieve effective cascading, MCCom leverages user actions as a novel signal to trigger the LLM only when the SLM fails, significantly reducing cloud computation costs. Furthermore, we introduce a two-stage speculative decoding strategy and an iterative retrieval mechanism to enhance collaboration between the models. We also train a 121M-parameter lightweight model, which achieves 73.8% of the performance of a 7B state-of-the-art model. Evaluated on RepoEval and a new real-world benchmark StmtEval, MCCom reduces inference latency by up to 47.9% and LLM usage by 46.3%, while improving the LLM's exact match rate by 8.9% through effective collaboration.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
