首页/论文/2007.03640

arXiv 2007.036402020-07-16cs.LGstat.ML

Benefiting Deep Latent Variable Models via Learning the Prior and Removing Latent Regularization

National Chiao Tung University（台湾交通大学）

Rogan Morrow, Wei-Chen Chiu

英文摘要：

There exist many forms of deep latent variable models, such as the variational autoencoder and adversarial autoencoder. Regardless of the specific class of model, there exists an implicit consensus that the latent distribution should be regularized towards the prior, even in the case where the prior distribution is learned. Upon investigating the effect of latent regularization on image generation our results indicate that in the case where a sufficiently expressive prior is learned, latent regularization is not necessary and may in fact be harmful insofar as image quality is concerned. We additionally investigate the benefit of learned priors on two common problems in computer vision: latent variable disentanglement, and diversity in image-to-image translation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies

Technion（以色列理工学院）；UIUC（伊利诺伊大学厄巴纳-香槟分校）

Many recent datasets contain a variety of different data modalities, for instance, image, question, and answer data in visual question answering (VQA). When training deep net classifiers on those multi-modal datasets, the modalities get exploited at different scales, i.e., some modalities can more easily contribute to the classification results than others. This is suboptimal because the classifier is inherently biased towards a subset of the modalities. To alleviate this shortcoming, we propose a novel regularization term based on the functional entropy. Intuitively, this term encourages to b…

2020-10-21cs.CVcs.LG

Sparsity in Variational Autoencoders

University of Bologna（博洛尼亚大学）

Working in high-dimensional latent spaces, the internal encoding of data in Variational Autoencoders becomes naturally sparse. We discuss this known but controversial phenomenon sometimes refereed to as overpruning, to emphasize the under-use of the model capacity. In fact, it is an important form of self-regularization, with all the typical benefits associated with sparsity: it forces the model to focus on the really important features, highly reducing the risk of overfitting. Especially, it is a major methodological guide for the correct tuning of the model capacity, progressively augmenting…

2019-02-13cs.LGstat.ML

Guillotine Regularization：为何在自监督学习中需要移除层以提升泛化能力

Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning

Meta AI Research（Meta人工智能研究院）；Mila, Université de Montréal（蒙特利尔大学米拉研究所）；Université Gustave Eiffel（古斯塔夫·埃菲尔大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）

该研究针对自监督学习中移除投影层反而提升下游泛化性能的现象，将其命名为Guillotine Regularization，揭示了其生效的根本原因，分析了最优层的影响因素，并给出了减少投影层需求的思路。

2023-06-09cs.LG

The Role of Regularization in Shaping Weight and Node Pruning Dependency and Dynamics

Ben-Gurion University（本-古里安大学）；Bar-Ilan University（巴伊兰大学）

The pressing need to reduce the capacity of deep neural networks has stimulated the development of network dilution methods and their analysis. While the ability of $L_1$ and $L_0$ regularization to encourage sparsity is often mentioned, $L_2$ regularization is seldom discussed in this context. We present a novel framework for weight pruning by sampling from a probability function that favors the zeroing of smaller weights. In addition, we examine the contribution of $L_1$ and $L_2$ regularization to the dynamics of node pruning while optimizing for weight pruning. We then demonstrate the effe…

2020-12-07cs.CVcs.LGeess.IV

BK-SDM：Stable Diffusion 的轻量、快速且低成本版本

BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusion

Nota Inc.（Nota公司）；Captions Research（Caption研究院）

本文提出 BK-SDM，通过 U-Net 块剪枝与特征蒸馏，在少量算力和数据下压缩 Stable Diffusion，保持零样本生成竞争力并支持边缘部署。

2024-12-02cs.LG

MaxDropout: Deep Neural Network Regularization Based on Maximum Output Values

Federal University of Sao Carlos - UFSCar（圣卡洛斯联邦大学）；Petróleo Brasileiro - Petrobras（巴西国家石油公司）；São Paulo State University - UNESP（圣保罗州立大学）

Different techniques have emerged in the deep learning scenario, such as Convolutional Neural Networks, Deep Belief Networks, and Long Short-Term Memory Networks, to cite a few. In lockstep, regularization methods, which aim to prevent overfitting by penalizing the weight connections, or turning off some units, have been widely studied either. In this paper, we present a novel approach called MaxDropout, a regularizer for deep neural network models that works in a supervised fashion by removing (shutting off) the prominent neurons (i.e., most active) in each hidden layer. The model forces fewe…

2020-07-27cs.LGstat.ML

更多相关论文收起更多论文

Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence

University of California, Los Angeles（加利福尼亚大学洛杉矶分校）

Regularization is typically understood as improving generalization by altering the landscape of local extrema to which the model eventually converges. Deep neural networks (DNNs), however, challenge this view: We show that removing regularization after an initial transient period has little effect on generalization, even if the final loss landscape is the same as if there had been no regularization. In some cases, generalization even improves after interrupting regularization. Conversely, if regularization is applied only after the initial transient, it has no effect on the final solution, who…

2019-05-30cs.AIcs.LGstat.ML

SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization

Kyung Hee University（庆熙大学）

Advanced data augmentation strategies have widely been studied to improve the generalization ability of deep learning models. Regional dropout is one of the popular solutions that guides the model to focus on less discriminative parts by randomly removing image regions, resulting in improved regularization. However, such information removal is undesirable. On the other hand, recent strategies suggest to randomly cut and mix patches and their labels among training images, to enjoy the advantages of regional dropout without having any pointless pixel in the augmented images. We argue that such r…

2021-07-27cs.LGstat.ML

突触修剪：一种启发自生物学的深度学习正则化方法

Synaptic Pruning: A Biological Inspiration for Deep Learning Regularization

College of Science and Engineering, James Cook University（詹姆斯库克大学科学与工程学院）；College of Health Care Sciences, James Cook University（詹姆斯库克大学健康护理科学学院）；College of Public Health, Medical, and Vet Sciences, James Cook University（詹姆斯库克大学公共卫生、医学与兽医学科学学院）

受生物突触修剪启发，提出基于权重幅值的渐进式动态修剪方法替代dropout正则化，无需单独微调阶段，在多模型多数据集上取得显著性能提升，尤其在金融时间序列预测中效果突出。

2025-10-05cs.AIcs.LG

One-dimensional Deep Image Prior for Time Series Inverse Problems

The University of Texas at Austin（得克萨斯大学奥斯汀分校）

We extend the Deep Image Prior (DIP) framework to one-dimensional signals. DIP is using a randomly initialized convolutional neural network (CNN) to solve linear inverse problems by optimizing over weights to fit the observed measurements. Our main finding is that properly tuned one-dimensional convolutional architectures provide an excellent Deep Image Prior for various types of temporal signals including audio, biological signals, and sensor measurements. We show that our network can be used in a variety of recovery tasks including missing value imputation, blind denoising, and compressed se…

2019-04-18cs.LGstat.ML

Learning Sparse Filters in Deep Convolutional Neural Networks with a l1/l2 Pseudo-Norm

Universite Clermont Auvergne（克莱蒙奥弗涅大学）；INSA Lyon（里昂国立应用科学学院）

While deep neural networks (DNNs) have proven to be efficient for numerous tasks, they come at a high memory and computation cost, thus making them impractical on resource-limited devices. However, these networks are known to contain a large number of parameters. Recent research has shown that their structure can be more compact without compromising their performance. In this paper, we present a sparsity-inducing regularization term based on the ratio l1/l2 pseudo-norm defined on the filter coefficients. By defining this pseudo-norm appropriately for the different filter kernels, and removing …

2020-07-20cs.LGcs.NE

Deep Generative Model using Unregularized Score for Anomaly Detection with Heterogeneous Complexity

Kobe University（神户大学）

Accurate and automated detection of anomalous samples in a natural image dataset can be accomplished with a probabilistic model for end-to-end modeling of images. Such images have heterogeneous complexity, however, and a probabilistic model overlooks simply shaped objects with small anomalies. This is because the probabilistic model assigns undesirably lower likelihoods to complexly shaped objects that are nevertheless consistent with set standards. To overcome this difficulty, we propose an unregularized score for deep generative models (DGMs), which are generative models leveraging deep neur…

2018-09-04cs.CVcs.LGstat.ML