首页/论文/2011.01136

arXiv 2011.011362020-11-03cs.CLcs.LG

Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation

University of Sheffield（谢菲尔德大学）；Utrecht University（乌得勒支大学）

Ruizhe Li, Xiao Li, Guanyi Chen, Chenghua Lin

备注：: Accepted by COLING 2020, final camera ready version

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

The Variational Autoencoder (VAE) is a popular and powerful model applied to text modelling to generate diverse sentences. However, an issue known as posterior collapse (or KL loss vanishing) happens when the VAE is used in text modelling, where the approximate posterior collapses to the prior, and the model will totally ignore the latent variables and be degraded to a plain language model during text generation. Such an issue is particularly prevalent when RNN-based VAE models are employed for text modelling. In this paper, we propose a simple, generic architecture called Timestep-Wise Regularisation VAE (TWR-VAE), which can effectively avoid posterior collapse and can be applied to any RNN-based VAE models. The effectiveness and versatility of our model are demonstrated in different tasks, including language modelling and dialogue response generation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Variational Autoencoders Without the Variation

University of Exeter（埃克塞特大学）；College of Engineering, Mathematics and Physical Sciences（工程、数学与物理科学学院）

Variational autoencdoers (VAE) are a popular approach to generative modelling. However, exploiting the capabilities of VAEs in practice can be difficult. Recent work on regularised and entropic autoencoders have begun to explore the potential, for generative modelling, of removing the variational approach and returning to the classic deterministic autoencoder (DAE) with additional novel regularisation methods. In this paper we empirically explore the capability of DAEs for image generation without additional novel methods and the effect of the implicit regularisation and smoothness of large ne…

2022-03-01cs.CVcs.LGstat.ML

A Stable Variational Autoencoder for Text Modelling

University of Aberdeen（阿伯丁大学）；University of Sheffield（谢菲尔德大学）

Variational Autoencoder (VAE) is a powerful method for learning representations of high-dimensional data. However, VAEs can suffer from an issue known as latent variable collapse (or KL loss vanishing), where the posterior collapses to the prior and the model will ignore the latent codes in generative tasks. Such an issue is particularly prevalent when employing VAE-RNN architectures for text modelling (Bowman et al., 2016). In this paper, we present a simple architecture called holistic regularisation VAE (HR-VAE), which can effectively avoid latent variable collapse. Compared to existing VAE…

2019-11-13cs.CLcs.LG

残差化时间稀疏自编码器用于解释扩散模型

Residualized Temporal Sparse Autoencoders for Interpreting Diffusion Models

University of California, Irvine（加州大学 Irvine 分校）

提出残差化时间稀疏自编码器，通过去噪时间步间的线性预测残差学习扩散激活轨迹中的可解释特征，并在Stable Diffusion 1.5上验证其有效性。

2026-10-08cs.AIcs.CVcs.LG

Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders

Imperial College London（帝国理工学院）

We study a variant of the variational autoencoder model (VAE) with a Gaussian mixture as a prior distribution, with the goal of performing unsupervised clustering through deep generative models. We observe that the known problem of over-regularisation that has been shown to arise in regular VAEs also manifests itself in our model and leads to cluster degeneracy. We show that a heuristic called minimum information constraint that has been shown to mitigate this effect in VAEs can also be applied to improve unsupervised clustering performance with our model. Furthermore we analyse the effect of …

2017-01-13cs.LGcs.NEstat.ML

Balancing Reconstruction Quality and Regularisation in ELBO for VAEs

University of Oxford（牛津大学）；Imperial College London（伦敦帝国学院）

A trade-off exists between reconstruction quality and the prior regularisation in the Evidence Lower Bound (ELBO) loss that Variational Autoencoder (VAE) models use for learning. There are few satisfactory approaches to deal with a balance between the prior and reconstruction objective, with most methods dealing with this problem through heuristics. In this paper, we show that the noise variance (often set as a fixed value) in the Gaussian likelihood p(x|z) for real-valued data can naturally act to provide such a balance. By learning this noise variance so as to maximise the ELBO loss, we auto…

2019-09-09cs.CVcs.LGstat.ML

SAeUron：基于稀疏自编码器的扩散模型可解释概念遗忘

SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoders

Bartosz Cywiński（独立研究者）；Kamil Deja（独立研究者）

SAeUron利用稀疏自编码器学习可解释特征，实现对扩散模型生成内容的概念遗忘，在UnlearnCanvas和I2P基准上优于现有方法，并能同时移除多概念且抵抗对抗攻击。

2025-05-21cs.AIcs.LG

更多相关论文收起更多论文

Constraining Variational Inference with Geometric Jensen-Shannon Divergence

University of Cambridge（剑桥大学）

We examine the problem of controlling divergences for latent space regularisation in variational autoencoders. Specifically, when aiming to reconstruct example $x\in\mathbb{R}^{m}$ via latent space $z\in\mathbb{R}^{n}$ ($n\leq m$), while balancing this against the need for generalisable latent representations. We present a regularisation mechanism based on the skew-geometric Jensen-Shannon divergence $\left(\textrm{JS}^{\textrm{G}_α}\right)$. We find a variation in $\textrm{JS}^{\textrm{G}_α}$, motivated by limiting cases, which leads to an intuitive interpolation between forward and reverse K…

2021-01-04cs.LGstat.ML

Regularising Inverse Problems with Generative Machine Learning Models

University of Bath（巴斯大学）

Deep neural network approaches to inverse imaging problems have produced impressive results in the last few years. In this paper, we consider the use of generative models in a variational regularisation approach to inverse problems. The considered regularisers penalise images that are far from the range of a generative model that has learned to produce images similar to a training dataset. We name this family \textit{generative regularisers}. The success of generative regularisers depends on the quality of the generative model and so we propose a set of desired criteria to assess generative mo…

2022-06-18cs.CVcs.LGeess.IV

使用正则化稀疏自编码器预测良好反应坐标与分子动力学轨迹的未来演化：一种新型深度学习方法

Prediction of good reaction coordinates and future evolution of MD trajectories using Regularized Sparse Autoencoders: A novel deep learning approach

本文提出用正则化稀疏自编码器从MD轨迹中自动发现少量关键反应坐标，并同时预测轨迹未来演化，在丙氨酸二肽和原黄素-DNA嵌入体系中验证了方法有效性。

2024-01-13cs.LGphysics.chem-phq-bio.QM

On PAC-Bayesian reconstruction guarantees for VAEs

University of Oxford（牛津大学）；University College London（伦敦大学学院）；Inria（法国国家信息与自动化研究所）

Despite its wide use and empirical successes, the theoretical understanding and study of the behaviour and performance of the variational autoencoder (VAE) have only emerged in the past few years. We contribute to this recent line of work by analysing the VAE's reconstruction ability for unseen test data, leveraging arguments from the PAC-Bayes theory. We provide generalisation bounds on the theoretical reconstruction error, and provide insights on the regularisation effect of VAE objectives. We illustrate our theoretical results with supporting experiments on classical benchmark datasets.

2022-02-23cs.CVcs.LGmath.ST

共享LoRA微调扩散模型权重时的风险

Risks When Sharing LoRA Fine-Tuned Diffusion Model Weights

University of Toronto（多伦多大学）

本文研究共享LoRA微调扩散模型权重的隐私泄露风险，提出带时间步嵌入的变分自编码器可从权重重建私有身份图像，且现有防御方法无法兼顾效用与隐私。

2024-09-13cs.CRcs.CVcs.LG

VAE解纠缠中重建损失被忽视的影响

Overlooked Implications of the Reconstruction Loss for VAE Disentanglement

University of the Witwatersrand（金山大学）

本文揭示VAE解纠缠主要源于数据与重建损失的交互而非正则化，并通过构造对抗数据集与定制重建损失验证了解纠缠的主观性。

2023-08-09cs.AIcs.CVcs.LG