Wavelets are well known for data compression, yet have rarely been applied to the compression of neural networks. This paper shows how the fast wavelet transform can be used to compress linear layers in neural networks. Linear layers still occupy a significant portion of the parameters in recurrent neural networks (RNNs). Through our method, we can learn both the wavelet bases and corresponding coefficients to efficiently represent the linear layers of RNNs. Our wavelet compressed RNNs have significantly fewer parameters yet still perform competitively with the state-of-the-art on synthetic and real-world RNN benchmarks. Wavelet optimization adds basis flexibility, without large numbers of extra weights. Source code is available at https://github.com/v0lta/Wavelet-network-compression.
High-Frequency (HF) signals are ubiquitous in the industrial world and are of great use for monitoring of industrial assets. Most deep learning tools are designed for inputs of fixed and/or very limited size and many successful applications of deep learning to the industrial context use as inputs extracted features, which is a manually and often arduously obtained compact representation of the original signal. In this paper, we propose a fully unsupervised deep learning framework that is able to extract a meaningful and sparse representation of raw HF signals. We embed in our architecture impo…
Despite the remarkable success of deep learning in pattern recognition, deep network models face the problem of training a large number of parameters. In this paper, we propose and evaluate a novel multi-path wavelet neural network architecture for image classification with far less number of trainable parameters. The model architecture consists of a multi-path layout with several levels of wavelet decompositions performed in parallel followed by fully connected layers. These decomposition operations comprise wavelet neurons with learnable parameters, which are updated during the training phas…
Signal denoising is a key preprocessing step for many applications, as the performance of a learning task is closely related to the quality of the input data. In this paper, we apply a signal processing based deep neural network architecture, a learnable extension of the wavelet packet transform. As main advantages, this model has few parameters, an intuitive initialization and strong learning capabilities. Moreover, we show that it is possible to easily modify the parameters of the model after the training step to tailor to different noise intensities. Two case studies are conducted to compar…
IIT Hyderabad（印度理工学院海得拉巴分校）；RWTH Aachen（亚琛工业大学）；Zerotha Research（泽罗塔研究公司）；Cerence Gmbh（赛轮思有限公司）；The University of Tokyo（东京大学）
Learning on evolving(dynamic) graphs has caught the attention of researchers as static methods exhibit limited performance in this setting. The existing methods for dynamic graphs learn spatial features by local neighborhood aggregation, which essentially only captures the low pass signals and local interactions. In this work, we go beyond current approaches to incorporate global features for effectively learning representations of a dynamically evolving graph. We propose to do so by capturing the spectrum of the dynamic graph. Since static methods to learn the graph spectrum would not conside…
Beyond Fixed Patches: Enhancing GPTs for Financial Prediction with Adaptive Segmentation and Learnable Wavelets
School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；School of Mathematical Sciences, Tongji University（同济大学数学科学学院）；Seek Data Group, Emoney Inc.（Seek Data集团，Emoney公司）
National Yang Ming Chiao Tung University（阳明交通大学）；University at Albany - SUNY（奥尔巴尼大学 - 纽约州立大学）；University at Buffalo（布法罗大学）
Pooling is a simple but essential layer in modern deep CNN architectures for feature aggregation and extraction. Typical CNN design focuses on the conv layers and activation functions, while leaving the pooling layers with fewer options. We introduce the Learning Discrete Wavelet Pooling (LDW-Pooling) that can be applied universally to replace standard pooling operations to better extract features with improved accuracy and efficiency. Motivated from the wavelet theory, we adopt the low-pass (L) and high-pass (H) filters horizontally and vertically for pooling on a 2D feature map. Feature sign…
Adaptive ship-radiated noise recognition with learnable fine-grained wavelet transform
Key Laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, Chinese Academy of Sciences（中国科学院声学研究所）；University of Chinese Academy of Sciences（中国科学院大学）
DB2-TransF: All You Need Is Learnable Daubechies Wavelets for Time Series Forecasting
organization= Department of Computer Science \& Engineering \& Communication \& Engineering , addressline= G B Pant , city= New Delhi , postcode= 580007 , state= Delhi , country= India；organization= Department of Computer Science \& Engineering , addressline= Indian Institute of Technology , city= Dharwad , postcode= 580007 , state= Karnataka , country= India
Convolutional Neural Networks (CNNs) are known for requiring extensive computational resources, and quantization is among the best and most common methods for compressing them. While aggressive quantization (i.e., less than 4-bits) performs well for classification, it may cause severe performance degradation in image-to-image tasks such as semantic segmentation and depth estimation. In this paper, we propose Wavelet Compressed Convolution (WCC) -- a novel approach for high-resolution activation maps compression integrated with point-wise convolutions, which are the main computational cost of m…
Université de Montréal（蒙特利尔大学）；University of California Los Angeles（加州大学洛杉矶分校）；Yale University（耶鲁大学）；Mila – the Quebec AI Institute（米拉-魁北克人工智能研究所）
We propose a new graph neural network (GNN) module, based on relaxations of recently proposed geometric scattering transforms, which consist of a cascade of graph wavelet filters. Our learnable geometric scattering (LEGS) module enables adaptive tuning of the wavelets to encourage band-pass features to emerge in learned representations. The incorporation of our LEGS-module in GNNs enables the learning of longer-range graph relations compared to many popular GNNs, which often rely on encoding graph structure via smoothness or similarity between neighbors. Further, its wavelet priors result in s…
Northeast Electric Power University（东北电力大学）；Technical University of Denmark（丹麦技术大学）
A fault diagnosis method for power electronics converters based on deep feedforward network and wavelet compression is proposed in this paper. The transient historical data after wavelet compression are used to realize the training of fault diagnosis classifier. Firstly, the correlation analysis of the voltage or current data running in various fault states is performed to remove the redundant features and the sampling point. Secondly, the wavelet transform is used to remove the redundant data of the features, and then the training sample data is greatly compressed. The deep feedforward networ…
Department of Computer Science, Purdue University（普渡大学计算机科学系）；Departments of Mathematics, Computer Science, Data and Decision Sciences, Technion（技术学院数学、计算机科学、数据与决策科学系）；Google Research（谷歌研究）
Graph Convolutional Networks (GCNs) are widely used in a variety of applications, and can be seen as an unstructured version of standard Convolutional Neural Networks (CNNs). As in CNNs, the computational cost of GCNs for large input graphs (such as large point clouds or meshes) can be high and inhibit the use of these networks, especially in environments with low computational resources. To ease these costs, quantization can be applied to GCNs. However, aggressive quantization of the feature maps can lead to a significant degradation in performance. On a different note, Haar wavelet transform…
Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compression
School of Mathematics, Harbin Institute of Technology, Harbin, P. R. China（哈尔滨工业大学数学学院，哈尔滨，中华人民共和国）；Shanghai Artificial Intelligence Laboratory, Shanghai, P. R. China（上海人工智能实验室，上海，中华人民共和国）
Cancer Research Center, University of Hawaii at Manoa（夏威夷大学马诺阿分校癌症研究中心）
We develop a neural network model to classify liver cancer patients into high-risk and low-risk groups using genomic data. Our approach provides a novel technique to classify big data sets using neural network models. We preprocess the data before training the neural network models. We first expand the data using wavelet analysis. We then compress the wavelet coefficients by mapping them onto a new scaled orthonormal coordinate system. Then the data is used to train a neural network model that enables us to classify cancer patients into two different classes of high-risk and low-risk patients.…
Multiresolution Matrix Factorization (MMF) is unusual amongst fast matrix factorization algorithms in that it does not make a low rank assumption. This makes MMF especially well suited to modeling certain types of graphs with complex multiscale or hierarchical strucutre. While MMF promises to yields a useful wavelet basis, finding the factorization itself is hard, and existing greedy methods tend to be brittle. In this paper we propose a learnable version of MMF that carfully optimizes the factorization with a combination of reinforcement learning and Stiefel manifold optimization through back…
ChWDTA: Channel-wise Wavelet-Domain Transformer Attention and Entropy Modeling for Learned Image Compression
Electrical and Computer Engineering Department, The University of British Columbia（英属哥伦比亚大学电气与计算机工程系）；School of Engineering Science, Simon Fraser University（西蒙弗雷泽大学工程科学学院）；School of Electronic Science and Technology, Eastern Institute of Technology（电子科学与技术学院，东部技术学院）；Google LLC（谷歌公司）
Yale University（耶鲁大学）；Mila -- Quebec AI Institute（米拉-魁北克人工智能研究所）
We propose a new graph neural network (GNN) module, based on relaxations of recently proposed geometric scattering transforms, which consist of a cascade of graph wavelet filters. Our learnable geometric scattering (LEGS) module enables adaptive tuning of the wavelets to encourage band-pass features to emerge in learned representations. The incorporation of our LEGS-module in GNNs enables the learning of longer-range graph relations compared to many popular GNNs, which often rely on encoding graph structure via smoothness or similarity between neighbors. Further, its wavelet priors result in s…
We propose a method to incrementally learn an embedding space over the domain of network architectures, to enable the careful selection of architectures for evaluation during compressed architecture search. Given a teacher network, we search for a compressed network architecture by using Bayesian Optimization (BO) with a kernel function defined over our proposed embedding space to select architectures for evaluation. We demonstrate that our search algorithm can significantly outperform various baseline methods, such as random search and reinforcement learning (Ashok et al., 2018). The compress…
Department of Physics（物理系）；The Hong Kong University of Science and Technology（香港科技大学）
Wavelet transformation stands as a cornerstone in modern data analysis and signal processing. Its mathematical essence is an invertible transformation that discerns slow patterns from fast ones in the frequency domain. Such an invertible transformation can be learned by a designed normalizing flow model. With a generalized lifting scheme as coupling layers, a factor-out layer resembling the downsampling, and parameter sharing at different levels of the model, one can train the normalizing flow to filter high-frequency elements at different levels, thus extending traditional linear wavelet tran…
RatioWaveNet: A Learnable RDWT Front-End for Robust and Interpretable EEG Motor-Imagery Classification
Department of Electrical, Electronics and Informatics Engineering, University of Catania（电子、电子与信息工程系，卡塔尼亚大学）；Department of Engineering, University of Palermo（工程系，巴勒莫大学）
We consider learning deep neural networks (DNNs) that consist of low-precision weights and activations for efficient inference of fixed-point operations. In training low-precision networks, gradient descent in the backward pass is performed with high-precision weights while quantized low-precision weights and activations are used in the forward pass to calculate the loss function for training. Thus, the gradient descent becomes suboptimal, and accuracy loss follows. In order to reduce the mismatch in the forward and backward passes, we utilize mean squared quantization error (MSQE) regularizat…
BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebook
The Hong Kong University of Science and Technology（香港科技大学）；City University of Hong Kong（香港城市大学）；University of Electronic Science and Technology of China（电子科技大学）
