The technologically-relevant task of feature extraction from data performed in deep-learning systems is routinely accomplished as repeated fast Fourier transforms (FFT) electronically in prevalent domain-specific architectures such as in graphics processing units (GPUs). However, electronics systems are limited with respect to power dissipation and delay, both, due to wire-charging challenges related to interconnect capacitance. Here we present a silicon photonics-based architecture for convolutional neural networks that harnesses the phase property of light to perform FFTs efficiently by executing the convolution as a multiplication in the Fourier-domain. The algorithmic executing time is determined by the time-of-flight of the signal through this photonic reconfigurable passive FFT filter circuit and is on the order of 10s of picosecond. A sensitivity analysis shows that this optical processor must be thermally phase stabilized corresponding to a few degrees. Furthermore, we find that for a small sample number, the obtainable number of convolutions per {time-power-chip area) outperforms GPUs by about 2 orders of magnitude. Lastly, we show that, conceptually, the optical FFT and convolution-processing performance is indeed directly linked to optoelectronic device-level, and improvements in plasmonics, metamaterials or nanophotonics are fueling next generation densely interconnected intelligent photonic circuits with relevance for edge-computing 5G networks.
The fast Fourier transform, FFT, is a useful and prevalent algorithm in signal processing. It characterizes the spectral components of a signal, or is used in combination with other operations to perform more complex computations such as filtering, convolution, and correlation. Digital FFTs are limited in speed by the necessity of moving charge within logic gates. An analog temporal FFT in fiber optics has been demonstrated with highest data bandwidth. However, the implementation with discrete fiber optic FFT components is bulky. Here, we present and analyze a design of an optical FFT in Silic…
For the benefit of designing scalable, fault resistant optical neural networks (ONNs), we investigate the effects architectural designs have on the ONNs' robustness to imprecise components. We train two ONNs -- one with a more tunable design (GridNet) and one with better fault tolerance (FFTNet) -- to classify handwritten digits. When simulated without any imperfections, GridNet yields a better accuracy (~98%) than FFTNet (~95%). However, under a small amount of error in their photonic components, the more fault tolerant FFTNet overtakes GridNet. We further provide thorough quantitative and qu…
With the proliferation of ultra-high-speed mobile networks and internet-connected devices, along with the rise of artificial intelligence, the world is generating exponentially increasing amounts of data - data that needs to be processed in a fast, efficient and smart way. These developments are pushing the limits of existing computing paradigms, and highly parallelized, fast and scalable hardware concepts are becoming progressively more important. Here, we demonstrate a computational specific integrated photonic tensor core - the optical analog of an ASIC-capable of operating at Tera-Multiply…
Convolution neural network (CNN), as one of the most powerful and popular technologies, has achieved remarkable progress for image and video classification since its invention in 1989. However, with the high definition video-data explosion, convolution layers in the CNN architecture will occupy a great amount of computing time and memory resources due to high computation complexity of matrix multiply accumulate operation. In this paper, a novel integrated photonic CNN is proposed based on double correlation operations through interleaved time-wavelength modulation. Micro-ring based multi-wavel…
An ultrafast single-pixel optical 2D imaging system using a single multimode fiber (MF) is proposed. The MF acted as the all-optical random pattern generator. Light with different wavelengths pass through a single MF will generator all-optical random speckle patterns, which have a low correlation of 0.074 with 0.1nm wavelength step from 1518.0nm to 1567.9nm. The all-optical random speckle patterns are perfect for compressive sensing (CS) imaging with the advantage of low cost in comparison with the conventional expensive pseudorandom binary sequence (PRBS). Besides, with the employment of phot…
In this work we numerically analyze a passive photonic integrated neuromorphic accelerator based on hardware-friendly optical spectrum slicing nodes. The proposed scheme can act as a fully analogue convolutional layer, preprocessing information directly in the optical domain. The proposed scheme allows the extraction of meaningful spatio-temporal features from the incoming data, thus when used prior to a simple fully connected digital single layer network it can boost performance with negligible power consumption. Numerical simulations using the MNIST dataset confirmed the acceleration propert…
With recent rapid advances in photonic integrated circuits, it has been demonstrated that programmable photonic chips can be used to implement artificial neural networks. Convolutional neural networks (CNN) are a class of deep learning methods that have been highly successful in applications such as image classification and speech processing. We present an architecture to implement a photonic CNN using the Fourier transform property of integrated star couplers. We show, in computer simulation, high accuracy image classification using the MNIST dataset. We also model component imperfections in …
Photonic technologies have shown a promising way to build high-speed and high-energy-efficiency neural network accelerators. In previously presented photonic neural networks, architectures are mainly designed for fully-connected layers. When convolutional layers are executed in such neural networks, the large-scale electrooptic modulation array heavily increases the energy dissipation on chip. To increase the energy efficiency, here we show an integrated photonic architecture specifically for convolutional layer calculations. Optical delay lines replace electronics to execute data manipulation…
Convolutional Neural Networks (CNNs) are powerful and highly ubiquitous tools for extracting features from large datasets for applications such as computer vision and natural language processing. However, a convolution is a computationally expensive operation in digital electronics. In contrast, neuromorphic photonic systems, which have experienced a recent surge of interest over the last few years, propose higher bandwidth and energy efficiencies for neural network training and inference. Neuromorphic photonics exploits the advantages of optical electronics, including the ease of analog proce…
Neuromorphic photonics has recently emerged as a promising hardware accelerator, with significant potential speed and energy advantages over digital electronics, for machine learning algorithms such as neural networks of various types. Integrated photonic networks are particularly powerful in performing analog computing of matrix-vector multiplication (MVM) as they afford unparalleled speed and bandwidth density for data transmission. Incorporating nonvolatile phase-change materials in integrated photonic devices enables indispensable programming and in-memory computing capabilities for on-chi…
Tensor analytics lays mathematical basis for the prosperous promotion of multiway signal processing. To increase computing throughput, mainstream processors transform tensor convolutions to matrix multiplications to enhance parallelism of computing. However, such order-reducing transformation produces data duplicates and consumes additional memory. Here, we demonstrate an integrated photonic tensor flow processor without tensor-matrix transformation, which outputs the convolved tensor as the input tensor 'flows' through the processor. The hybrid manipulation of optical dimensions of wavelength…
All-optical binary convolution with a photonic spiking vertical-cavity surface-emitting laser (VCSEL) neuron is proposed and demonstrated experimentally for the first time. Optical inputs, extracted from digital images and temporally encoded using rectangular pulses, are injected in the VCSEL neuron which delivers the convolution result in the number of fast (<100 ps long) spikes fired. Experimental and numerical results show that binary convolution is achieved successfully with a single spiking VCSEL neuron and that all-optical binary convolution can be used to calculate image gradient magnit…
Convolutional neural networks (CNNs), inspired by biological visual cortex systems, are a powerful category of artificial neural networks that can extract the hierarchical features of raw data to greatly reduce the network parametric complexity and enhance the predicting accuracy. They are of significant interest for machine learning tasks such as computer vision, speech recognition, playing board games and medical diagnosis. Optical neural networks offer the promise of dramatically accelerating computing speed to overcome the inherent bandwidth bottleneck of electronics. Here, we demonstrate …
The parallelism of optics and the miniaturization of optical components using nanophotonic structures, such as metasurfaces present a compelling alternative to electronic implementations of convolutional neural networks. The lack of a low-power optical nonlinearity, however, requires slow and energy-inefficient conversions between the electronic and optical domains. Here, we design an architecture which utilizes a single electrical to optical conversion by designing a free-space optical frontend unit that implements the linear operations of the first layer with the subsequent layers realized e…
Photonic waveguides are prime candidates for integrated and parallel photonic interconnects. Such interconnects correspond to large-scale vector matrix products, which are at the heart of neural network computation. However, parallel interconnect circuits realized in two dimensions, for example by lithography, are strongly limited in size due to disadvantageous scaling. We use three dimensional (3D) printed photonic waveguides to overcome this limitation. 3D optical-couplers with fractal topology efficiently connect large numbers of input and output channels, and we show that the substrate's f…
Programmable electro-optic frequency comb empowers integrated parallel convolution processing
State Key Laboratory of Optoelectronic Materials and Technologies, School of Electronics and Information Technology, Sun Yat-sen University（中山大学电子与信息工程学院光电材料与技术国家重点实验室）；State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University（清华大学精密仪器系精密测量技术与仪器国家重点实验室）
Designing modern photonic devices often involves traversing a large parameter space via an optimization procedure, gradient based or otherwise, and typically results in the designer performing electromagnetic simulations of correlated devices. In this paper, we present an approach to accelerate the Generalized Minimal Residual (GMRES) algorithm for the solution of frequency-domain Maxwell's equations using two machine learning models (principal component analysis and a convolutional neural network) trained on simulations of correlated devices. These data-driven models are trained to predict a …
This paper introduces the concept of on-chip temporal optical computing, based on dispersive Fourier transform and suitably designed modulation module, to perform mathematical operations of interest, such as differentiation, integration, or convolution in time domain. The desired mathematical operation is performed as signal propagates through a fully reconfigurable on-chip photonic signal processor. Although a few number of photonic temporal signal processors have been introduced recently, they are usually bulky or they suffer from limited reconfigurability which is of great importance to imp…
