arXiv 1803.085772018-03-22stat.MLcs.LG

Unbiased scalable softmax optimization

Department of Industrial Engineering and Operations Research（工业工程与运筹学系）；Columbia University（哥伦比亚大学）

Francois Fagan, Garud Iyengar

英文摘要：

Recent neural network and language models rely on softmax distributions with an extremely large number of categories. Since calculating the softmax normalizing constant in this context is prohibitively expensive, there is a growing literature of efficiently computable but biased estimates of the softmax. In this paper we propose the first unbiased algorithms for maximizing the softmax likelihood whose work per iteration is independent of the number of classes and datapoints (and no extra work is required at the end of each epoch). We show that our proposed unbiased methods comprehensively outperform the state-of-the-art on seven real world datasets.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

One-vs-Each Approximation to Softmax for Scalable Estimation of Probabilities

The softmax representation of probabilities for categorical variables plays a prominent role in modern machine learning with numerous applications in areas such as large scale classification, neural language modeling and recommendation systems. However, softmax estimation is very expensive for large scale inference because of the high cost associated with computing the normalizing constant. Here, we introduce an efficient approximation to softmax probabilities which takes the form of a rigorous lower bound on the exact probability. This bound is expressed as a product over pairwise probabiliti…

2016-10-29stat.ML

具有通用小批量一致性与无偏全集梯度近似的可扩展集合编码

Scalable Set Encoding with Universal Mini-Batch Consistency and Unbiased Full Set Gradient Approximation

KAIST（韩国科学技术院）；National University of Singapore（新加坡国立大学）

本文提出 UMBC 集合函数类与恒定内存、无偏全集梯度近似的 MBC 训练算法，增强可扩展集合编码的表达能力，并在多类任务中验证其效率与效果。

2023-06-08cs.LGstat.ML

Extreme Classification via Adversarial Softmax Approximation

University of California, Irvine（加州大学尔湾分校）；Department of Computer Science（计算机科学系）

Training a classifier over a large number of classes, known as 'extreme classification', has become a topic of major interest with applications in technology, science, and e-commerce. Traditional softmax regression induces a gradient cost proportional to the number of classes $C$, which often is prohibitively expensive. A popular scalable softmax approximation relies on uniform negative sampling, which suffers from slow convergence due a poor signal-to-noise ratio. In this paper, we propose a simple training method for drastically enhancing the gradient signal by drawing negative samples from …

2020-02-15cs.LGstat.ML

Unbiased Subdata Selection for Fair Classification: A Unified Framework and Scalable Algorithms

Virginia Tech（弗吉尼亚理工大学）

As an important problem in modern data analytics, classification has witnessed varieties of applications from different domains. Different from conventional classification approaches, fair classification concerns the issues of unintentional biases against the sensitive features (e.g., gender, race). Due to high nonconvexity of fairness measures, existing methods are often unable to model exact fairness, which can cause inferior fair classification outcomes. This paper fills the gap by developing a novel unified framework to jointly optimize accuracy and fairness. The proposed framework is vers…

2020-12-24cs.LGstat.ML

Enabling scalable stochastic gradient-based inference for Gaussian processes by employing the Unbiased LInear System SolvEr (ULISSE)

In applications of Gaussian processes where quantification of uncertainty is of primary interest, it is necessary to accurately characterize the posterior distribution over covariance parameters. This paper proposes an adaptation of the Stochastic Gradient Langevin Dynamics algorithm to draw samples from the posterior distribution over covariance parameters with negligible bias and without the need to compute the marginal likelihood. In Gaussian process regression, this has the enormous advantage that stochastic gradients can be computed by solving linear systems only. A novel unbiased linear …

2015-09-03stat.COstat.MEstat.ML

FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models

University of Toronto（多伦多大学）；Vector Institute（向量研究所）；OpenAI

A promising class of generative models maps points from a simple distribution to a complex distribution through an invertible neural network. Likelihood-based training of these models requires restricting their architectures to allow cheap computation of Jacobian determinants. Alternatively, the Jacobian trace can be used if the transformation is specified by an ordinary differential equation. In this paper, we use Hutchinson's trace estimator to give a scalable unbiased estimate of the log-density. The result is a continuous-time invertible generative model with unbiased density estimation an…

2018-10-22cs.CVcs.LGstat.ML

更多相关论文收起更多论文

面向可扩展神经隐式采样器的基于熵的训练方法

Entropy-based Training Methods for Scalable Neural Implicit Sampler

School of Mathematical Sciences（数学科学学院）；Peking University（北京大学）；Academy for Advanced Interdisciplinary Studies（先进跨学科研究学院）

本文提出一种可扩展的神经隐式采样器，通过神经变换直接生成大批量样本，并引入KL训练和Fisher训练两种方法优化采样器，在三个不同规模的采样基准上验证了其有效性、效率和可扩展性。

2025-06-05cs.LGstat.ML

Scalable Kernel Methods via Doubly Stochastic Gradients

Georgia Institute of Technology（佐治亚理工学院）；Carnegie Mellon University（卡内基梅隆大学）

The general perception is that kernel methods are not scalable, and neural nets are the methods of choice for nonlinear learning problems. Or have we simply not tried hard enough for kernel methods? Here we propose an approach that scales up kernel methods using a novel concept called "doubly stochastic functional gradients". Our approach relies on the fact that many kernel methods can be expressed as convex optimization problems, and we solve the problems by making two unbiased stochastic approximations to the functional gradient, one using random training points and another using random func…

2015-09-10cs.LGstat.ML

Fine-grained Uncertainty Modeling in Neural Networks

National University of Singapore（新加坡国立大学）；Untangle AI（Untangle人工智能公司）

Existing uncertainty modeling approaches try to detect an out-of-distribution point from the in-distribution dataset. We extend this argument to detect finer-grained uncertainty that distinguishes between (a). certain points, (b). uncertain points but within the data distribution, and (c). out-of-distribution points. Our method corrects overconfident NN decisions, detects outlier points and learns to say ``I don't know'' when uncertain about a critical point between the top two predictions. In addition, we provide a mechanism to quantify class distributions overlap in the decision manifold and…

2020-02-11cs.CVcs.LGstat.ML

Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers

Google（谷歌）；University of Cambridge（剑桥大学）；Alan Turing Institute（艾伦·图灵研究所）；DeepMind（深度思维）

Transformer models have achieved state-of-the-art results across a diverse range of domains. However, concern over the cost of training the attention mechanism to learn complex dependencies between distant inputs continues to grow. In response, solutions that exploit the structure and sparsity of the learned attention matrix have blossomed. However, real-world applications that involve long sequences, such as biological sequence analysis, may fall short of meeting these assumptions, precluding exploration of these models. To address this challenge, we present a new Transformer architecture, Pe…

2020-10-01cs.CLcs.LGstat.ML

可扩展贝叶斯蒙特卡洛：超越深度集成的快速不确定性估计

Scalable Bayesian Monte Carlo: fast uncertainty estimation beyond deep ensembles

Mathematics Department, University of Manchester（曼彻斯特大学数学系）；Department of Electrical and Computer Engineering, Southern Methodist University（南方 Methodist 大学电气与计算机工程系）；DEVCOM US Army Research Laboratory（美国陆军研发实验室）；Department of Electrical and Computer Engineering, University of Illinois Chicago（伊利诺伊大学芝加哥分校电气与计算机工程系）；Cognizant AI Labs（Cognizant AI 实验室）；Quantum Information Science Section, Oak Ridge National Laboratory（橡树岭国家实验室量子信息科学部门）；Tulane University（Tulane 大学）

本文提出可扩展贝叶斯蒙特卡洛（SBMC）方法，通过并行SMC或MCMC算法在点估计与后验之间插值，在相同计算时间内取得与深度集成相当或更好的精度，并显著提升认知不确定性量化能力。

2025-08-21cs.LGstat.COstat.ML

Training recurrent networks online without backtracking

We introduce the "NoBackTrack" algorithm to train the parameters of dynamical systems such as recurrent neural networks. This algorithm works in an online, memoryless setting, thus requiring no backpropagation through time, and is scalable, avoiding the large computational and memory cost of maintaining the full gradient of the current state with respect to the parameters. The algorithm essentially maintains, at each time, a single search direction in parameter space. The evolution of this search direction is partly stochastic and is constructed in such a way to provide, at every time, an unbi…

2015-11-20cs.LGcs.NEstat.ML

可扩展离散扩散采样器：组合优化与统计物理

Scalable Discrete Diffusion Samplers: Combinatorial Optimization and Statistical Physics

ELLIS Unit Linz, LIT AI Lab, Johannes Kepler University Linz, Austria（林茨ELLIS单元、LIT AI实验室、林茨约瑟夫·冯·海德里希大学，奥地利）；Department of Electrical & Computer Engineering, Georgia Institute of Technology（电气与计算机工程系，佐治亚理工学院）；NXAI Lab & NXAI GmbH, Linz, Austria（NXAI实验室及NXAI GmbH，林茨，奥地利）

本文提出两种内存高效的离散扩散采样器训练方法（基于策略梯度和自归一化神经重要性采样），实现无偏采样，在组合优化和伊辛模型基准上超越自回归方法。

2025-07-08cond-mat.stat-mechcs.AIcs.LG