arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.00501cs.CLcs.AIcs.PF
收藏

BaseRT: 通过原生Metal在Apple Silicon上实现最佳LLM推理

BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal

Base Compute, Melbourne, Australia(Base Compute,墨尔本,澳大利亚)

Prabod Rathnayaka, Fabian Waschkowski, Lukas Wesemann

AI总结:

提出基于原生Metal的推理运行时BaseRT,通过芯片特定内核融合、统一内存感知优化和自定义调度逻辑,在Apple Silicon上实现最高推理吞吐量,相比现有框架提升高达1.56倍解码吞吐量。

摘要:

我们提出BaseRT,一个针对Apple Silicon上大型语言模型(LLM)的原生Metal推理运行时,并报告了迄今为止该硬件上最高的推理吞吐量。现有运行时,包括this http URL和基于MLX的框架,由于抽象层未针对Metal的执行模型或Apple Silicon的统一内存拓扑设计而产生开销。通过基于Metal原生构建,结合芯片特定内核融合、统一内存感知优化和自定义调度逻辑,BaseRT恢复了基于框架的方法所遗留的性能。BaseRT支持所有Apple M系列设备上的八种量化格式(Q2至FP16)的广泛模型系列。在本文中,我们在M3和M4 Pro设备上评估了Qwen3、Llama 3.2和Gemma 4系列在Q4和Q8量化下的性能。BaseRT相比this http URL实现了高达1.56倍的解码吞吐量提升,相比MLX实现了高达1.35倍的提升,在混合专家模型的预填充阶段优势更为显著,为从小于1B到30B参数的模型提供了一致的同类最佳吞吐量。这些结果表明,Apple Silicon是一个比先前报道的更强大的推理平台,对新兴的边缘推理范式具有直接影响:随着隐私要求、延迟约束和云成本压力推动推理向设备端部署发展,性能优化的本地运行时是实现这一过渡的关键使能层。BaseRT可在以下网址公开获取:this https URL

英文摘要:

We present BaseRT, a native Metal inference runtime for large language models (LLMs) on Apple Silicon, and report the highest inference throughput on this hardware to date. Existing runtimes, including llama.cpp and MLX-based frameworks, incur overhead from abstractions not designed for Metal's execution model or Apple Silicon's unified memory topology. By building natively on Metal with chip-specific kernel fusion, unified memory-aware optimisation, and custom dispatch logic, BaseRT recovers performance that framework-based approaches leave on the table. BaseRT supports a wide range of model families across eight quantisation formats (Q2 to FP16) on all Apple M-series devices. In this paper, we evaluate the Qwen3, Llama 3.2, and Gemma 4 families at Q4 and Q8 quantisation on M3 and M4 Pro devices. BaseRT achieves up to 1.56x higher decode throughput than llama.cpp and up to 1.35x higher than MLX, with substantially larger margins on prefill for mixture-of-experts models, delivering consistent best-in-class throughput from sub-1B to 30B parameter models. These results establish Apple Silicon as a more capable inference platform than previously reported, with direct implications for the emerging edge inference paradigm: as privacy requirements, latency constraints, and cloud cost pressures drive inference toward on-device deployment, performance-optimised local runtimes are a critical enabling layer for this transition. BaseRT is publicly available at https://github.com/basecompute/baseRT

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑