LLM-based agents deliver state-of-the-art performance across tasks but incur high end-to-end latency on edge devices. We introduce Agent-X, a software-only, accuracy-preserving framework that accelerates both the prefill and decode stages of on-device agent workloads. Agent-X's two key components rewrite prompts to leverage prefix caching tailored to agent-specific input-token patterns and enable LLM-free speculative decoding for fast token generation with minimal overhead. On representative agentic workloads, Agent-X achieves a 1.61x end-to-end speedup in real systems with no accuracy loss and can be seamlessly integrated into existing on-device AI agents. To the best of our knowledge, ours is the first to systematically characterize and eliminate latency bottlenecks in on-device agents.
HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipeline
School of Integrated Circuits, Peking University(北京大学集成电路学院);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);Beijing Advanced Innovation Center for Integrated Circuits(北京集成电路高精尖创新中心)
本文提出 HG-PIPE 流水线 FPGA 加速器,通过混合粒度架构、数据流与并行耦合及 LUT 近似消除气泡并降低资源开销,显著提升 ViT 吞吐与能效。
ASTRA: Communication-Efficient Acceleration for Multi-Device Transformer Inference
Manning College of Information and Computer Sciences, University of Massachusetts Amherst, MA, US(信息与计算机科学学院,马萨诸塞大学阿默斯特分校,马萨诸塞州,美国);Amazon, Seattle, WA, US(亚马逊,华盛顿州西雅图,美国);Amazon AWS, Washington, D.C., US(亚马逊AWS,华盛顿特区,美国)
Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devices
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);Data Science and Analytics Thrust, HKUST (GZ)(香港科技大学(广州)数据科学与分析学域);The Chinese University of Hong Kong(香港中文大学)
NNStreamer: Efficient and Agile Development of On-Device AI Systems
Samsung Research, Samsung Electronics(三星电子三星研究院)
我们提出了NNStreamer,一个软件系统,处理神经网络作为流管道的过滤器,应用流处理范式的深度神经网络应用。随着深度神经网络应用的广泛普及,一个新趋势是设备上人工智能。它是在移动设备或边缘/物联网设备上处理神经网络,而不是云服务器。新出现的隐私问题、数据传输成本和运营成本意味着对设备上AI的需求,特别是如果我们部署了大量设备。NNStreamer可有效处理设备上具有复杂数据流管道的神经网络,以最小的努力显著提高整体性能。此外,NNStreamer简化了实现,并允许直接重用现成的媒体过滤器,这大大降低了开发成本。我们已经在广泛的产品和平台上部署了NNStreamer,包括Galaxy系列和各种消费电子设备。实验结果表明,在开发成本和提高性能的管道架构和NNStreamer的减少。它是由Linux Foundation AI孵化的开源项目,可向公众提供,适用于各种硬件和软件平台。
Hermes: Memory-Efficient Pipeline Inference for Large Models on Edge Devices
ParisTech Elite Institute of Technology, Shanghai Jiao Tong University(上海交通大学巴黎卓越工程师学院);School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);UM-SJTU Joint Institute, Shanghai Jiao Tong University(上海交通大学密西根学院);Shanghai Aerospace System Engineering Institute(上海航天系统工程研究所);The University of Melbourne(墨尔本大学)
Fully Learnable Group Convolution for Acceleration of Deep Neural Networks
Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心)
A Privacy-Preserving-Oriented DNN Pruning and Mobile Acceleration Framework
Northeastern University(东北大学);The College of William and Mary(威廉与玛丽学院);University of Connecticut(康涅狄格大学);University of Illinois at Chicago(伊利诺伊大学芝加哥分校)
TriADA: Massively Parallel Trilinear Matrix-by-Tensor Multiply-Add Algorithm and Device Architecture for the Acceleration of 3D Discrete Transformations