Parallel convolution processing using an integrated photonic tensor core
Johannes Feldmann, Nathan Youngblood, Maxim Karpov, Helge Gehring, Xuan Li, Maik Stappers, Manuel Le Gallo, Xin Fu, Anton Lukashchuk, Arslan Raja, Junqiu Liu, D… 展开作者
Johannes Feldmann, Nathan Youngblood, Maxim Karpov, Helge Gehring, Xuan Li, Maik Stappers, Manuel Le Gallo, Xin Fu, Anton Lukashchuk, Arslan Raja, Junqiu Liu, David Wright, Abu Sebastian, Tobias Kippenberg, Wolfram Pernice, Harish Bhaskaran
With the proliferation of ultra-high-speed mobile networks and internet-connected devices, along with the rise of artificial intelligence, the world is generating exponentially increasing amounts of data - data that needs to be processed in a fast, efficient and smart way. These developments are pushing the limits of existing computing paradigms, and highly parallelized, fast and scalable hardware concepts are becoming progressively more important. Here, we demonstrate a computational specific integrated photonic tensor core - the optical analog of an ASIC-capable of operating at Tera-Multiply-Accumulate per second (TMAC/s) speeds. The photonic core achieves parallelized photonic in-memory computing using phase-change memory arrays and photonic chip-based optical frequency combs (soliton microcombs). The computation is reduced to measuring the optical transmission of reconfigurable and non-resonant passive components and can operate at a bandwidth exceeding 14 GHz, limited only by the speed of the modulators and photodetectors. Given recent advances in hybrid integration of soliton microcombs at microwave line rates, ultra-low loss silicon nitride waveguides, and high speed on-chip detectors and modulators, our approach provides a path towards full CMOS wafer-scale integration of the photonic tensor core. While we focus on convolution processing, more generally our results indicate the major potential of integrated photonics for parallel, fast, and efficient computational hardware in demanding AI applications such as autonomous driving, live video processing, and next generation cloud computing services.
The technologically-relevant task of feature extraction from data performed in deep-learning systems is routinely accomplished as repeated fast Fourier transforms (FFT) electronically in prevalent domain-specific architectures such as in graphics processing units (GPUs). However, electronics systems are limited with respect to power dissipation and delay, both, due to wire-charging challenges related to interconnect capacitance. Here we present a silicon photonics-based architecture for convolutional neural networks that harnesses the phase property of light to perform FFTs efficiently by exec…
Convolution neural network (CNN), as one of the most powerful and popular technologies, has achieved remarkable progress for image and video classification since its invention in 1989. However, with the high definition video-data explosion, convolution layers in the CNN architecture will occupy a great amount of computing time and memory resources due to high computation complexity of matrix multiply accumulate operation. In this paper, a novel integrated photonic CNN is proposed based on double correlation operations through interleaved time-wavelength modulation. Micro-ring based multi-wavel…
Today's unrelenting increase in demand for information processing creates the need for novel computing concepts. Reservoir computing is such a concept that lends itself particularly well to photonic hardware implementations. Over recent years, these hardware implementations have gained maturity and now achieve state-of-the-art performance on several benchmark tasks. However, implementations so far are essentially all based on sequential data processing, leaving the inherent parallelism of photonics unexploited. Parallel implementations process all neurons simultaneously, and therefore have the…
It is believed that neural information representation and processing relies on the neural population instead of a single neuron. In neuromorphic photonics, photonic neurons in the form of nonlinear responses have been extensively studied in single devices and temporal nodes. However, to construct a photonic neural population (PNP), the process of scaling up and massive interconnections remain challenging considering the physical complexity and response latency. Here, we propose a comb-based PNP interconnected by carrier coupling with superior scalability. Two unique properties of neural popula…
Saitama University（埼玉大学）；The University of Tokyo（东京大学）
Photonic accelerators have attracted increasing attention in artificial intelligence applications. The multi-armed bandit problem is a fundamental problem of decision making using reinforcement learning. However, the scalability of photonic decision making has not yet been demonstrated in experiments, owing to technical difficulties in physical realization. We propose a parallel photonic decision-making system for solving large-scale multi-armed bandit problems using optical spatiotemporal chaos. We solve a 512-armed bandit problem online, which is much larger than previous experiments by two …
In this work we numerically analyze a passive photonic integrated neuromorphic accelerator based on hardware-friendly optical spectrum slicing nodes. The proposed scheme can act as a fully analogue convolutional layer, preprocessing information directly in the optical domain. The proposed scheme allows the extraction of meaningful spatio-temporal features from the incoming data, thus when used prior to a simple fully connected digital single layer network it can boost performance with negligible power consumption. Numerical simulations using the MNIST dataset confirmed the acceleration propert…
With recent rapid advances in photonic integrated circuits, it has been demonstrated that programmable photonic chips can be used to implement artificial neural networks. Convolutional neural networks (CNN) are a class of deep learning methods that have been highly successful in applications such as image classification and speech processing. We present an architecture to implement a photonic CNN using the Fourier transform property of integrated star couplers. We show, in computer simulation, high accuracy image classification using the MNIST dataset. We also model component imperfections in …
Photonic technologies have shown a promising way to build high-speed and high-energy-efficiency neural network accelerators. In previously presented photonic neural networks, architectures are mainly designed for fully-connected layers. When convolutional layers are executed in such neural networks, the large-scale electrooptic modulation array heavily increases the energy dissipation on chip. To increase the energy efficiency, here we show an integrated photonic architecture specifically for convolutional layer calculations. Optical delay lines replace electronics to execute data manipulation…
In this paper, we have proposed the compound structure of one-dimensional photonic crystal (PC), which includes series connection and parallel connection PC. We have studied the transmission characteristics of series connection and parallel connection PC, and obtained some new results. In addition, we have proved the series connection one-dimensional PC can realize the logical AND operation, and the parallel connection one-dimensional PC can realize the logical OR operation. The compound structure of one-dimensional PC can design more new type structure optical devices, and will provide the ba…
Convolutional Neural Networks (CNNs) are powerful and highly ubiquitous tools for extracting features from large datasets for applications such as computer vision and natural language processing. However, a convolution is a computationally expensive operation in digital electronics. In contrast, neuromorphic photonic systems, which have experienced a recent surge of interest over the last few years, propose higher bandwidth and energy efficiencies for neural network training and inference. Neuromorphic photonics exploits the advantages of optical electronics, including the ease of analog proce…
Neuromorphic photonics has recently emerged as a promising hardware accelerator, with significant potential speed and energy advantages over digital electronics, for machine learning algorithms such as neural networks of various types. Integrated photonic networks are particularly powerful in performing analog computing of matrix-vector multiplication (MVM) as they afford unparalleled speed and bandwidth density for data transmission. Incorporating nonvolatile phase-change materials in integrated photonic devices enables indispensable programming and in-memory computing capabilities for on-chi…
The recent rapid increase in demand for data processing has resulted in the need for novel machine learning concepts and hardware. Physical reservoir computing and an extreme learning machine are novel computing paradigms based on physical systems themselves, where the high dimensionality and nonlinearity play a crucial role in the information processing. Herein, we propose the use of multidimensional speckle dynamics in multimode fibers for information processing, where input information is mapped into the space, frequency, and time domains by an optical phase modulation technique. The speckl…
Tensor analytics lays mathematical basis for the prosperous promotion of multiway signal processing. To increase computing throughput, mainstream processors transform tensor convolutions to matrix multiplications to enhance parallelism of computing. However, such order-reducing transformation produces data duplicates and consumes additional memory. Here, we demonstrate an integrated photonic tensor flow processor without tensor-matrix transformation, which outputs the convolved tensor as the input tensor 'flows' through the processor. The hybrid manipulation of optical dimensions of wavelength…
Reconfigurable photonic mesh networks of tunable beamsplitter nodes can linearly transform $N$-dimensional vectors representing input modal amplitudes of light for applications such as energy-efficient machine learning hardware, quantum information processing, and mode demultiplexing. Such photonic meshes are typically programmed and/or calibrated by tuning or characterizing each beam splitter one-by-one, which can be time-consuming and can limit scaling to larger meshes. Here we introduce a graph-topological approach that defines the general class of feedforward networks commonly used in such…
College of Future Information Technology, Fudan University（复旦大学未来信息技术学院）；Institute of Microelectronics, Agency for Science, Technology and Research（新加坡科技研究局微电子研究所）；Zhangjiang Laboratory（张江实验室）；Shanghai Innovation Institute（上海创新研究院）；State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）
The incorporation of multiplexing techniques used in Microwave Photonics to Quantum Key Distribution (QKD) systems bring important advantages enabling the simultaneous and parallel delivery of multiple keys between a central station and different end-users in the context of multipoint access and metropolitan networks, or by providing higher key distribution rates in point to point links by suitably linking the parallel distributed keys. It also allows the coexistence of classical information and quantum key distribution channels over a single optical fibre infrastructure. Here we show, for the…
Photonic waveguides are prime candidates for integrated and parallel photonic interconnects. Such interconnects correspond to large-scale vector matrix products, which are at the heart of neural network computation. However, parallel interconnect circuits realized in two dimensions, for example by lithography, are strongly limited in size due to disadvantageous scaling. We use three dimensional (3D) printed photonic waveguides to overcome this limitation. 3D optical-couplers with fractal topology efficiently connect large numbers of input and output channels, and we show that the substrate's f…
Univ. Bourgogne Franche-Comté（勃艮第-弗朗什-孔泰大学）；Technische Universität Berlin（柏林工业大学）
Neural networks are one of the disruptive computing concepts of our time. However, they fundamentally differ from classical, algorithmic computing in a number of fundamental aspects. These differences result in equally fundamental, severe and relevant challenges for neural network computing using current computing substrates. Neural networks urge for parallelism across the entire processor and for a co-location of memory and arithmetic, i.e. beyond von Neumann architectures. Parallelism in particular made photonics a highly promising platform, yet until now scalable and integratable concepts a…
Photonic networks are considered a promising substrate for high-performance future computing systems. Compared to electronics, photonics has significant advantages for a fully parallel implementation of networks. A promising approach for parallel large-scale photonic networks is realizing the connections using diffraction. Here, we characterize the scalability of such diffractive coupling in great detail. Based on experiments, analytically obtained bounds and numerical simulations considering real-world optical imaging setups, we find that the concept in principle enables networks hosting over…
All-optical binary convolution with a photonic spiking vertical-cavity surface-emitting laser (VCSEL) neuron is proposed and demonstrated experimentally for the first time. Optical inputs, extracted from digital images and temporally encoded using rectangular pulses, are injected in the VCSEL neuron which delivers the convolution result in the number of fast (<100 ps long) spikes fired. Experimental and numerical results show that binary convolution is achieved successfully with a single spiking VCSEL neuron and that all-optical binary convolution can be used to calculate image gradient magnit…
Parallel Spatial Photonic Programming of Optoelectronic IGZO RRAM with a compact $μ$LED Array
Institute of Photonics, SUPA Dept. of Physics, University of Strathclyde（斯特拉斯克莱德大学光子学研究所、SUPA物理系）；Institute for Integrated Micro and Nano Systems, University of Edinburgh（爱丁堡大学集成微纳系统研究所）
Photonic Neural Network implementations have been gaining considerable attention as a potentially disruptive future technology. Demonstrating learning in large scale neural networks is essential to establish photonic machine learning substrates as viable information processing systems. Realizing photonic Neural Networks with numerous nonlinear nodes in a fully parallel and efficient learning hardware was lacking so far. We demonstrate a network of up to 2500 diffractively coupled photonic nodes, forming a large scale Recurrent Neural Network. Using a Digital Micro Mirror Device, we realize rei…
Quantum Scattering Model of Dual Parallel Mach Zehnder Electro Optic Modulators: General Formalism, Impairments, and Applications to Frequency-Bin Photonic Quantum States
Universitat Politècnica de València（瓦伦西亚理工大学）；Universidad Miguel Hernández de Elche（埃尔切米格尔·埃尔南德斯大学）；Universidad de Vigo（维戈大学）
Convolutional neural networks (CNNs), inspired by biological visual cortex systems, are a powerful category of artificial neural networks that can extract the hierarchical features of raw data to greatly reduce the network parametric complexity and enhance the predicting accuracy. They are of significant interest for machine learning tasks such as computer vision, speech recognition, playing board games and medical diagnosis. Optical neural networks offer the promise of dramatically accelerating computing speed to overcome the inherent bandwidth bottleneck of electronics. Here, we demonstrate …
A novel parallel optical delay detector (PODD) is proposed for angle-of-arrival (AOA) measurement of a microwave signal with selective measurement range and signal-to-noise (SNR) enhancement. The PODD is experimentally demonstrated by using a dual parallel Mach-Zehnder modulator. The noise robustness enhancement is experimentally investigated. Based on the proposed PODD, the measurement error for the full range can be less than 0.02 radian (0.71% of full measured range).
The inability of conventional electronic architectures to efficiently solve large combinatorial problems motivates the development of novel computational hardware. There has been much effort recently toward developing novel, application-specific hardware, across many different fields of engineering, such as integrated circuits, memristors, and photonics. However, unleashing the true potential of such novel architectures requires the development of featured algorithms which optimally exploit their fundamental properties. We here present the Photonic Recurrent Ising Sampler (PRIS), a heuristic m…
In this paper, we have proposed a new compound structure one-dimensional photonic crystal, which include series connection, parallel connection and positive and negative feedback compound structure photonic crystal. We have studied their transmission characteristics and obtained some new results, which should be help to design new type optical devices, such as optical amplifier, photonic crystal laser and so on.
The parallelism of optics and the miniaturization of optical components using nanophotonic structures, such as metasurfaces present a compelling alternative to electronic implementations of convolutional neural networks. The lack of a low-power optical nonlinearity, however, requires slow and energy-inefficient conversions between the electronic and optical domains. Here, we design an architecture which utilizes a single electrical to optical conversion by designing a free-space optical frontend unit that implements the linear operations of the first layer with the subsequent layers realized e…
The optical domain is a promising field for physical implementation of neural networks, due to the speed and parallelism of optics. Extreme Learning Machines (ELMs) are feed-forward neural networks in which only output weights are trained, while internal connections are randomly selected and left untrained. Here we report on a photonic ELM based on a frequency-multiplexed fiber setup. Multiplication by output weights can be performed either offline on a computer, or optically by a programmable spectral filter. We present both numerical simulations and experimental results on classification tas…
We extend a modal theory of diffraction by a set of parallel fibers to deal with the case of a hard boundary: that is a structure made for instance of air-holes inside a dielectric matrix. Numerical examples are given concerning some resonant phenomena.
Residue number system (RNS) enables dimensionality reduction of an arithmetic problem by representing a large number as a set of smaller integers, where the number is decomposed by prime number factorization using the moduli as basic functions. These reduced problem sets can then be processed independently and in parallel, thus improving computational efficiency and speed. Here we show an optical RNS hardware representation based on integrated nanophotonics. The digit-wise shifting in RNS arithmetic is expressed as spatial routing of an optical signal in 2x2 hybrid photonic-plasmonic switches.…
George Washington University（乔治华盛顿大学）；University of California, Los Angeles（加州大学洛杉矶分校）
Convolutional neural networks are paramount in image and signal processing including the relevant classification and training tasks alike and constitute for the majority of machine learning compute demand today. With convolution operations being computationally intensive, next generation hardware accelerators need to offer parallelization and algorithmic-hardware homomorphism. Fortunately, diffractive display optics is capable of million-channel parallel data processing at low latency, however, thus far only showed tens of Hertz slow single image and kernel capability, thereby significantly un…
A silicon inverse diamond structure whose lattice point shape was vacant regular octahedrons had a complete photonic band gap at around 0.4 THz and X point's photonic band gap (0.36 THz to 0.44 THz) by plane wave expansion method. It is said that three-dimensional photonic crystals have no polarization anisotropy in photonic band gap (stop gap, stop band) of high symmetry points in normal incidence. Experimental results, however, confirmed that the polarization orientation (electric-field direction) of a reflected wave was different from that of an incident wave whose direction was [001]. The …
Self-Configuring Universal Multichannel and Multidimensional Integrated Photonic Processing Engine
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Guangdong-Macao Joint Laboratory for Modular Chip Design and Testing, The Hong Kong University of Science and Technology (Guangzhou)（模块化芯片设计与测试粤港联合实验室，香港科技大学（广州））；Department of Electrical and Electronic Engineering, The University of Hong Kong（电气电子工程学系，香港大学）
Programmable electro-optic frequency comb empowers integrated parallel convolution processing
State Key Laboratory of Optoelectronic Materials and Technologies, School of Electronics and Information Technology, Sun Yat-sen University（中山大学电子与信息工程学院光电材料与技术国家重点实验室）；State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University（清华大学精密仪器系精密测量技术与仪器国家重点实验室）
We investigate the electromagnetic propagation in two-dimensional photonic crystals, formed by parallel dielectric cylinders embedded a uniform medium. The frequency band structure is computed using the standard plane-wave expansion method, while the propagation and scattering of the electromagnetic waves are calculated by the multiple scattering theory. It is shown that within partial bandgaps, the waves tend to bend away from the forbidden directions. Such a property may render novel applications in manipulating optical flows. In addition, the relevance with the imaging by flat photonic crys…
