Understanding and Mitigating Exploding Inverses in Invertible Neural Networks

Stabilizing Invertible Neural Networks Using Mixture Models

In this paper, we analyze the properties of invertible neural networks, which provide a way of solving inverse problems. Our main focus lies on investigating and controlling the Lipschitz constants of the corresponding inverse networks. Without such an control, numerical simulations are prone to errors and not much is gained against traditional approaches. Fortunately, our analysis indicates that changing the latent distribution from a standard normal one to a Gaussian mixture model resolves the issue of exploding Lipschitz constants. Indeed, numerical simulations confirm that this modificatio…

UnICORNN: A recurrent model for learning very long time dependencies

The design of recurrent neural networks (RNNs) to accurately process sequential inputs with long-time dependencies is very challenging on account of the exploding and vanishing gradient problem. To overcome this, we propose a novel RNN architecture which is based on a structure preserving discretization of a Hamiltonian system of second-order ordinary differential equations that models networks of oscillators. The resulting RNN is fast, invertible (in time), memory efficient and we derive rigorous bounds on the hidden state gradients to prove the mitigation of the exploding and vanishing gradi…

缓解LLMs在可逆问题中的幻觉和遗漏：应用于硬件逻辑设计自动化

Mitigating hallucinations and omissions in LLMs for invertible problems: An application to hardware logic design automation

本文提出利用LLMs作为无损编码器和解码器处理可逆问题，以减少LLMs的幻觉和遗漏，应用于硬件逻辑设计自动化，提升设计效率并检测错误。

递归神经网络：梯度消失与梯度爆炸并非故事的终点

Recurrent neural networks: vanishing and exploding gradients are not the end of the story

本文发现RNN优化困难不仅源于梯度消失/爆炸，还源于记忆增强时参数变化引发输出剧变，并揭示逐元素递归与精细参数化（如SSM、LSTM）可缓解此效应。

使用单步 Q-learning 缓解 Actor-Critic 方法中的离策略偏差：一种新型校正方法

Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approach

针对连续控制中 Actor-Critic 离策略学习的分布失配问题，提出基于策略相似度度量的单步 Q-learning 校正方法，实现安全离策略训练并以更少步数取得更高回报。

A Projection Algorithm for the Unitary Weights

Unitary neural networks are promising alternatives for solving the exploding and vanishing activation/gradient problem without the need for explicit normalization that reduces the inference speed. However, they often require longer training time due to the additional unitary constraints on their weight matrices. Here we show a novel algorithm using a backpropagation technique with Lie algebra for computing approximated unitary weights from their pre-trained, non-unitary counterparts. The unitary networks initialized with these approximations can reach the desired accuracies much faster, mitiga…