In this paper, the echo state network (ESN) memory capacity, which represents the amount of input data an ESN can store, is analyzed for a new type of deep ESNs. In particular, two deep ESN architectures are studied. First, a parallel deep ESN is proposed in which multiple reservoirs are connected in parallel allowing them to average outputs of multiple ESNs, thus decreasing the prediction error. Then, a series architecture ESN is proposed in which ESN reservoirs are placed in cascade that the output of each ESN is the input of the next ESN in the series. This series ESN architecture can capture more features between the input sequence and the output sequence thus improving the overall prediction accuracy. Fundamental analysis shows that the memory capacity of parallel ESNs is equivalent to that of a traditional shallow ESN, while the memory capacity of series ESNs is smaller than that of a traditional shallow ESN.In terms of normalized root mean square error, simulation results show that the parallel deep ESN achieves 38.5% reduction compared to the traditional shallow ESN while the series deep ESN achieves 16.8% reduction.
Scaling up Echo-State Networks with multiple light scattering
Laboratoire Kastler Brossel（卡斯特勒布罗塞尔实验室）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；Laboratoire de Physique Statistique（统计物理实验室）；LightOn（LightOn公司）
Echoes of the past: A unified perspective on fading memory and echo states
Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore（数学科学系，物理与数学科学学院，南洋理工大学，新加坡）；Nanyang Technological University, Singapore（南洋理工大学，新加坡）
Echo: A Large Language Model with Temporal Episodic Memory
Institute of AI Education, East China Normal University（华东师范大学人工智能教育学院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）
