First-Generation Inference Accelerator Deployment at Facebook
Michael Anderson, Benny Chen, Stephen Chen, Summer Deng, Jordan Fix, Michael Gschwind, Aravind Kalaiah, Changkyu Kim, Jaewon Lee, Jason Liang, Haixin Liu, Yingh… 展开作者
Michael Anderson, Benny Chen, Stephen Chen, Summer Deng, Jordan Fix, Michael Gschwind, Aravind Kalaiah, Changkyu Kim, Jaewon Lee, Jason Liang, Haixin Liu, Yinghai Lu, Jack Montgomery, Arun Moorthy, Satish Nadathur, Sam Naghshineh, Avinash Nayak, Jongsoo Park, Chris Petersen, Martin Schatz, Narayanan Sundaram, Bangsheng Tang, Peter Tang, Amy Yang, Jiecao Yu, Hector Yuen, Ying Zhang, Aravind Anbudurai, Vandana Balan, Harsha Bojja, Joe Boyd, Matthew Breitbach, Claudio Caldato, Anna Calvo, Garret Catron, Sneh Chandwani, Panos Christeas, Brad Cottel, Brian Coutinho, Arun Dalli, Abhishek Dhanotia, Oniel Duncan, Roman Dzhabarov, Simon Elmir, Chunli Fu, Wenyin Fu, Michael Fulthorp, Adi Gangidi, Nick Gibson, Sean Gordon, Beatriz Padilla Hernandez, Daniel Ho, Yu-Cheng Huang, Olof Johansson, Shishir Juluri, Shobhit Kanaujia, Manali Kesarkar, Jonathan Killinger, Ben Kim, Rohan Kulkarni, Meghan Lele, Huayu Li, Huamin Li, Yueming Li, Cynthia Liu, Jerry Liu, Bert Maher, Chandra Mallipedi, Seema Mangla, Kiran Kumar Matam, Jubin Mehta, Shobhit Mehta, Christopher Mitchell, Bharath Muthiah, Nitin Nagarkatte, Ashwin Narasimha, Bernard Nguyen, Thiara Ortiz, Soumya Padmanabha, Deng Pan, Ashwin Poojary, Ye, Qi, Olivier Raginel, Dwarak Rajagopal, Tristan Rice, Craig Ross, Nadav Rotem, Scott Russ, Kushal Shah, Baohua Shan, Hao Shen, Pavan Shetty, Krish Skandakumaran, Kutta Srinivasan, Roshan Sumbaly, Michael Tauberg, Mor Tzur, Sidharth Verma, Hao Wang, Man Wang, Ben Wei, Alex Xia, Chenyu Xu, Martin Yang, Kai Zhang, Ruoxi Zhang, Ming Zhao, Whitney Zhao, Rui Zhu, Ajit Mathews, Lin Qiao, Misha Smelyanskiy, Bill Jia, Vijay Rao
In this paper, we provide a deep dive into the deployment of inference accelerators at Facebook. Many of our ML workloads have unique characteristics, such as sparse memory accesses, large model sizes, as well as high compute, memory and network bandwidth requirements. We co-designed a high-performance, energy-efficient inference accelerator platform based on these requirements. We describe the inference accelerator platform ecosystem we developed and deployed at Facebook: both hardware, through Open Compute Platform (OCP), and software framework and tooling, through Pytorch/Caffe2/Glow. A characteristic of this ecosystem from the start is its openness to enable a variety of AI accelerators from different vendors. This platform, with six low-power accelerator cards alongside a single-socket host CPU, allows us to serve models of high complexity that cannot be easily or efficiently run on CPUs. We describe various performance optimizations, at both platform and accelerator level, which enables this platform to serve production traffic at Facebook. We also share deployment challenges, lessons learned during performance optimization, as well as provide guidance for future inference hardware co-design.
A3C3: AI Algorithm and Accelerator Co-design, Co-search, and Co-generation
Siebel School of Computing and Data Science(塞比尔计算与数据科学学院);Department of Electrical and Computer Engineering(电气与计算机工程系);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Eventor: An Efficient Event-Based Monocular Multi-View Stereo Accelerator on FPGA Platform
Beihang University(北京航空航天大学);Xidian University(西安电子科技大学);Beijing Real Imaging Medical Technology Co., Ltd.(北京真实影像医疗科技有限公司);The Chinese University of Hong Kong(香港中文大学)
Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators
Department of Computer Science(计算机科学系);San Francisco State University(旧金山州立大学);Argonne National Laboratory(阿贡国家实验室);Lawrence Berkeley National Laboratory(伯克利国家实验室)
AnalogNAS: A Neural Network Design Framework for Accurate Inference with Analog In-Memory Computing
Univ. Polytechnique Hauts-de-France(上法兰西理工大学);IBM Research Europe(IBM欧洲研究院);IBM T. J. Watson Research Center(IBM托马斯·J·沃森研究中心);IBM Research Almaden(IBM阿尔马登研究中心)
Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers
Kaze Technologies(凯兹技术);Kaze Consulting(凯兹咨询);Electrical and Computer Engineering(电气与计算机工程);Lebanese American University(黎巴嫩美国大学);Carnegie Mellon University(卡内基梅隆大学);University of Exeter(埃克塞特大学)