Unsupervised Learning in Echo State Networks for Input Reconstruction
Graduate School of Information Science and Technology, The University of Tokyo(东京大学信息科学与技术研究生院);School of Systems Information Science, Future University Hakodate(函馆未来大学系统信息科学系)
Taiki Yamada, Yuichi Katori, Kantaro Fujiwara
备注:
35 pages, 11 figures. This paper has been accepted for publication in Neural Computation (MIT Press)
回声状态网络(Echo State Networks,ESNs)是一类仅输出层可训练、循环层与输入层固定的循环神经网络。这种架构约束使其能够对时序数据进行计算高效的处理。传统上,ESN的输出层采用有监督学习结合目标输出进行训练。本研究聚焦于输入重构(Input Reconstruction,IR)任务,即训练输出层以重构输入ESN的输入时序数据。我们证明,只要ESN参数是先验已知的且满足可逆性条件,无需访问有监督目标即可通过无监督学习(Unsupervised Learning,UL)实现输入重构。该公式使得依赖输入重构的应用(如动力系统复制与噪声滤波)可通过与现有算法直接集成,在无监督学习框架下重新构建。我们的结果表明,ESN参数的先验知识可降低对监督的依赖,从而确立了一项新原则:不仅可通过固定部分网络参数,还可通过利用其具体取值来实现。此外,我们提出的用于输入重构及相关任务的基于无监督学习的算法适用于自主处理,为类似计算机制原则上如何在大脑中运作提供了见解。这些发现有助于更深入地理解ESN的数学基础及其与计算神经科学模型的相关性。
英文摘要:
Echo state networks (ESNs) are a class of recurrent neural networks in which only the readout layer is trainable, while the recurrent and input layers are fixed. This architectural constraint enables computationally efficient processing of time-series data. Traditionally, the readout layer in ESNs is trained using supervised learning with target outputs. In this study, we focus on input reconstruction (IR), where the readout layer is trained to reconstruct the input time series fed into the ESN. We show that IR can be achieved through unsupervised learning (UL), without access to supervised targets, provided that the ESN parameters are known a priori and satisfy invertibility conditions. This formulation allows applications relying on IR, such as dynamical system replication and noise filtering, to be reformulated within the UL framework via straightforward integration with existing algorithms. Our results suggest that prior knowledge of ESN parameters can reduce reliance on supervision, thereby establishing a new principle: not only by fixing part of the network parameters but also by exploiting their specific values. Furthermore, our UL-based algorithms for input reconstruction and related tasks are suitable for autonomous processing, offering insights into how analogous computational mechanisms might operate in the brain in principle. These findings contribute to a deeper understanding of the mathematical foundations of ESNs and their relevance to models in computational neuroscience.
EchoTab: Learning Measurement-Grounded Representations for Echocardiographic Diagnosis through Tabular Reconstruction
Technical University of Munich (TUM)(慕尼黑工业大学);TUM University Hospital(慕尼黑工业大学医院);Imperial College London(伦敦帝国理工学院);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心)
Unsupervised 4D Cardiac Motion Tracking with Spatiotemporal Optical Flow Networks
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);DJI Technology Co., Ltd.(大疆创新科技有限公司);Shenzhen University of Advanced Technology(深圳理工大学)
Model-based Reconstruction with Learning: From Unsupervised to Supervised and Beyond
Michigan State University(密歇根州立大学);University of Michigan - Shanghai Jiao Tong University Joint Institute(密歇根大学-上海交通大学联合学院);Shanghai Jiao Tong University(上海交通大学)
On Instabilities of Unsupervised Denoising Diffusion Models in Magnetic Resonance Imaging Reconstruction
University Hospital Aachen(亚琛大学医院);Else Kroener Fresenius Center for Digital Health(埃尔泽·克勒纳·弗雷森纽斯数字健康中心);Medical Faculty Carl Gustav Carus(卡尔·古斯塔夫·卡鲁斯医学院);Technische Universität Dresden(德累斯顿工业大学);University Hospital Dresden(德累斯顿大学医院);National Center for Tumor Diseases (NCT)(国家肿瘤疾病中心);University Hospital Heidelberg(海德堡大学医院)
Unsupervised Learning on a DIET: Datum IndEx as Target Free of Self-Supervision, Reconstruction, Projector Head
Meta AI;Cranberry-Lemon University(蔓越莓柠檬大学);University of the Witwatersrand(威特沃特斯兰德大学)
如果我们希望学习廉价、可靠和可转移的模型,那么代价高昂、嘈杂和过度专业化的标签将被搁置一边,转而支持无监督学习。为此,谱嵌入、自监督学习或生成式建模提供了有竞争力的解决方案。然而,这些方法带来了许多挑战,\textit{e.g.}估计测地线距离,指定投影仪架构和抗崩溃损失,或指定解码器架构和重建损失。相比之下,我们引入了一个简单的可解释的替代方案-创造的\textbf{DIET} -从未标记的数据中学习表示,没有这些挑战。\textbf{DIET}非常简单:采用自己喜欢的分类设置,并使用\textbf{D} atum \textbf{I} nd \textbf{E} x作为其\textbf{T}目标类,\textit{i.e. each sample is its own class},无需进一步更改。\textbf{DIET}不需要解码器/投影器网络,不基于正对也不基于重构,不引入超参数,并且可以跨数据集和架构进行开箱即用。尽管\textbf{DIET}很简单,但学习到的表示具有高质量,并且通常与最先进的\textit{e.g.}相当,在DIET的学习表示之上使用线性分类器,在CIFAR 100上使用Resnet 101达到$71.4\%$,在TinyImagenet上使用Resnext 50达到$52.5\%$。
Enhancing Low-dose CT Image Reconstruction by Integrating Supervised and Unsupervised Learning
University of Michigan Shanghai Jiao Tong University Joint Institute(密歇根大学上海交通大学联合学院);Shanghai Jiao Tong University(上海交通大学);Amazon Inc.(亚马逊公司);Michigan State University(密歇根州立大学)
Monitoring Post-Disaster Urban Recovery Using High-Resolution SAR Time Series and Unsupervised Learning: Evidence from the 2023 Türkiye-Syria Earthquake
Department of Electrical, Computer and Biomedical Engineering, University of Pavia(帕维亚大学电气、计算机和生物医学工程系);Department of Engineering, University of Sannio(桑尼奥大学工程系);Italian Space Agency (ASI)(意大利航天局)
Primitive Representation Learning for Unsupervised Dynamic Contrast Enhanced MRI Reconstruction
Helmholtz Munich(亥姆霍兹慕尼黑);Technical University of Munich(慕尼黑工业大学);Boston Children’s Hospital(波士顿儿童医院);Harvard Medical School(哈佛医学院);Munich Center for Machine Learning(慕尼黑机器学习中心);Imperial College London(伦敦帝国理工学院);King’s College London(伦敦国王学院)
Unsupervised denoising for sparse multi-spectral computed tomography
University of Oulu(奥卢大学);University College London(伦敦大学学院)
具有光子计数探测器(PCD)的多能量计算机断层扫描(CT)能够实现光谱成像,因为PCD可以将入射光子分配给特定的能量通道。然而,具有许多光谱通道的PCD大幅增加了CT重建的计算复杂度,并且定制重建算法需要根据不同的噪声统计进行微调。\rev{Especially if many projections are taken, a large amount of data has to be collected and stored. Sparse view CT is one solution for data reduction. However, these issues are especially exacerbated when sparse imaging scenarios are encountered due to a significant reduction in photon counts.}在这项工作中,我们研究了基于学习的改进,以获得高质量的重建从稀疏测量的64通道PCD-CT的挑战性任务的适用性。特别是,为了克服训练过程中丢失的参考数据,我们提出了一种无监督的去噪和伪影去除方法,通过在重建中利用不同的滤波器函数以及光谱通道与核范数的显式耦合。性能评估模拟合成数据和公开提供的实验多光谱成像通过计算机断层扫描(MUSIC)数据集。我们将我们的无监督方法的质量…
CogGen: Cognitive-Load-Inspired Fully Unsupervised Deep Generative Modeling for Compressively Sampled MRI Reconstruction
Medical AI Research Centre, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院医学人工智能研究中心);School of Biomedical Engineering, Southern Medical University(南方医科大学生物医学工程学院);School of Mathematics and Statistics, Xi’an Jiaotong University(西安交通大学数学与统计学院);Paul C. Lauterbur Research Centre for Biomedical Imaging, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院Paul C. Lauterbur生物医学成像研究中心)
Unsupervised Adaptive Neural Network Regularization for Accelerated Radial Cine MRI
Charité - Universitätsmedizin Berlin(夏里特大学医学院);Berlin Institute of Health(柏林健康研究所);Physikalisch-Technische Bundesanstalt (PTB)(德国联邦物理技术研究院);King’s College London(伦敦国王学院);Technical University of Berlin(柏林工业大学);University of Innsbruck(因斯布鲁克大学)
Accurate parameter estimation using scan-specific unsupervised deep learning for relaxometry and MR fingerprinting
Tsinghua University(清华大学);Zhejiang University(浙江大学);Harvard Medical School(哈佛医学院);Athinoula A. Martinos Center for Biomedical Imaging(阿西诺拉·A·马蒂诺斯生物医学成像中心);Korea Advanced Institute of Science and Technology(韩国科学技术院)
Unsupervised Spatial-spectral Hyperspectral Image Reconstruction and Clustering with Diffusion Geometry
City University of Hong Kong(香港城市大学);Hong Kong Centre for Cerebro-Cardiovascular Health Engineering(香港脑心血管健康工程中心);Tufts University(塔夫茨大学);Wake Forest University(维克森林大学)
Multi-feature Reconstruction Network using Crossed-mask Restoration for Unsupervised Industrial Anomaly Detection
School of Information and Communication Engineering, Zhongyuan University of Technology(中原工学院信息与通信工程学院);College of Automation Engineering, Nanjing University of Aeronautics and Astronautics(南京航空航天大学自动化学院);Applied Technology Research Institute, Henan Institute of Economics and Trade(河南经贸职业学院应用技术研究院)
MADGAN: unsupervised Medical Anomaly Detection GAN using multiple adjacent brain MRI slice reconstruction
LPIXEL Inc.(LPIXEL公司);University of Cambridge(剑桥大学);National Institute of Informatics(国立信息学研究所);National Center for Global Health and Medicine(国立国际医疗研究中心);Juntendo University(顺天堂大学);ELTE Eötvös Loránd University(厄特沃什·罗兰大学);The University of Tokyo(东京大学)
School of Biomedical Engineering, Shanghai Jiao Tong University(上海交通大学生物医学工程学院);School of Information Science and Technology, ShanghaiTech University(上海交通大学信息科学与技术学院);School of Biomedical Engineering and the National Engineering Research Center of Advanced Magnetic Resonance Technologies for Diagnosis and Therapy (NERCAMRT), Shanghai Jiao Tong University(上海交通大学生物医学工程学院和先进磁共振诊断与治疗国家工程研究中心)