Light-weight convolutional neural networks (CNNs) have small complexity and are good candidates for low-power, high-throughput inference. Such networks are heterogeneous in terms of computation-to-communication (CTC) ratios and computation patterns between layers, especially for different layer types. Yet, existing AI processors either use homogeneous processing elements (PEs), resulting in low runtime PE efficiency, or run different layers on heterogeneous PEs in sequential, introducing resource redundancy. This paper proposes a heterogeneous dual-core architecture (dual-OPU), where one core is optimized for regular convolution layers and the other for depthwise convolution layers. PEs are homogeneous with each core. To make full use of dual-core parallelism, we develop a scheduling algorithm to concurrently execute layers for different input images on dual-core and balance parallel workload. Meanwhile, we automatically tune the PE number for a core and tune the input size for each PE to maximize throughput. Compared with a single-core processor with the same area for a single network, heterogeneous dual-OPU on average improves runtime PE efficiency and throughput by 11% and 31%, respectively. For a workload of multiple networks, dual-OPU improves average throughput by 11% compared with the state-of-the-art processors scaled to the same area. To the best of our knowledge, it is the first in-depth study on the heterogeneous dual-core processor for light-weight CNNs.
SkipOPU: An FPGA-based Overlay Processor for Large Language Models with Dynamically Allocated Computation
University of California, Los Angeles(加州大学洛杉矶分校);Eastern Institute of Technology(宁波诺丁汉大学);Institute of Digital Twin, Eastern Institute of Technology(宁波诺丁汉大学数字孪生研究院);Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Institute of Digital Twin, Eastern Institute of Technology(宁波诺丁汉大学数字孪生研究院空间智能与数字衍生宁波市重点实验室);Chiplet CAD and Manufacturing Engineering Research Center of Zhejiang Province, Ningbo Institute of Digital Twin, Eastern Institute of Technology(浙江省芯片级封装CAD与制造工程研究中心,宁波诺丁汉大学数字孪生研究院)
Trinity College Dublin, The University of Dublin(都柏林大学圣三一学院);The School of Computer Science and Statistics(计算机科学与统计学院);Institute of Technology Carlow(卡洛理工学院)