In this work, we optimize speculative sampling for parallel hardware accelerators to improve sampling speed. We notice that substantial portions of the intermediate matrices necessary for speculative sampling can be computed concurrently. This allows us to distribute the workload across multiple GPU threads, enabling simultaneous operations on matrix segments within thread blocks. This results in profiling time improvements ranging from 6% to 13% relative to the baseline implementation, without compromising accuracy. To further accelerate speculative sampling, probability distributions parameterized by softmax are approximated by sigmoid. This approximation approach results in significantly greater relative improvements in profiling time, ranging from 37% to 94%, with a minor decline in accuracy. We conduct extensive experiments on both automatic speech recognition and summarization tasks to validate the effectiveness of our optimization methods.
EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China(软件研究所智能软件研究中心,中国科学院,北京);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京)
A GPU-Outperforming FPGA Accelerator Architecture for Binary Convolutional Neural Networks
Arizona State University(亚利桑那州立大学);Nanyang Technological University(南洋理工大学)
基于FPGA的卷积神经网络(CNN)硬件加速器由于其比GPU更高的能量效率而受到广泛关注。然而,基于FPGA的解决方案要实现比GPU更高的吞吐量是具有挑战性的。在本文中,我们证明了当CNN在权重和激活的二进制约束下进行训练时,FPGA加速在吞吐量和能效方面都是一种优越的解决方案。具体来说,我们提出了一种优化的FPGA加速器架构,专为逐位卷积和归一化而设计,具有大规模的空间并行性和深流水线阶段。FPGA加速器的一个关键优势是其性能对数据批量大小不敏感,而GPU加速的性能在很大程度上取决于数据的批量大小。实验结果表明,在Virtex-7 FPGA上运行的二进制CNN的加速器架构比Titan X GPU快8.3倍,能效高75倍,可以处理小批量的在线个人请求。对于处理大批量静态数据,该解决方案在吞吐量方面与Titan X GPU不相上下,同时提供9.5倍的能效。
AccUDNN: A GPU Memory Efficient Accelerator for Training Ultra-deep Neural Networks
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)