首页/论文/2007.05447

arXiv 2007.054472020-07stat.MLcs.LG

Generalized Maximum Entropy for Supervised Classification

BCAM-Basque Center for Applied Mathematics（巴斯克应用数学中心）；IKERBASQUE-Basque Foundation for Science（巴斯克科学基金会）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）

Santiago Mazuelas, Yuan Shen, Aritz Pérez

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

The maximum entropy principle advocates to evaluate events' probabilities using a distribution that maximizes entropy among those that satisfy certain expectations' constraints. Such principle can be generalized for arbitrary decision problems where it corresponds to minimax approaches. This paper establishes a framework for supervised classification based on the generalized maximum entropy principle that leads to minimax risk classifiers (MRCs). We develop learning techniques that determine MRCs for general entropy functions and provide performance guarantees by means of convex optimization. In addition, we describe the relationship of the presented techniques with existing classification methods, and quantify MRCs performance in comparison with the proposed bounds and conventional methods.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Semi-Supervised Learning with Declaratively Specified Entropy Constraints

Carnegie Mellon University（卡内基梅隆大学）；Tencent AI Lab（腾讯人工智能实验室）

We propose a technique for declaratively specifying strategies for semi-supervised learning (SSL). The proposed method can be used to specify ensembles of semi-supervised learning, as well as agreement constraints and entropic regularization constraints between these learners, and can be used to model both well-known heuristics such as co-training and novel domain-specific heuristics. In addition to representing individual SSL heuristics, we show that multiple heuristics can also be automatically combined using Bayesian optimization methods. We show consistent improvements on a suite of well-s…

2018-05-18cs.AIcs.LGstat.ML

Deep Transductive Semi-supervised Maximum Margin Clustering

Semi-supervised clustering is an very important topic in machine learning and computer vision. The key challenge of this problem is how to learn a metric, such that the instances sharing the same label are more likely close to each other on the embedded space. However, little attention has been paid to learn better representations when the data lie on non-linear manifold. Fortunately, deep learning has led to great success on feature learning recently. Inspired by the advances of deep learning, we propose a deep transductive semi-supervised maximum margin clustering approach. More specifically…

2015-01-26cs.LG

Information-theoretic Semi-supervised Metric Learning via Entropy Regularization

Tokyo Institute of Technology（东京工业大学）；Purdue University（普渡大学）

We propose a general information-theoretic approach called Seraph (SEmi-supervised metRic leArning Paradigm with Hyper-sparsity) for metric learning that does not rely upon the manifold assumption. Given the probability parameterized by a Mahalanobis distance, we maximize the entropy of that probability on labeled data and minimize it on unlabeled data following entropy regularization, which allows the supervised and unsupervised parts to be integrated in a natural and meaningful way. Furthermore, Seraph is regularized by encouraging a low-rank projection induced from the metric. The optimizat…

2012-06-18cs.LGstat.ML

Semi-supervised cross-entropy clustering with information bottleneck constraint

Jagiellonian University（雅盖隆大学）；Technical University of Munich（慕尼黑工业大学）

In this paper, we propose a semi-supervised clustering method, CEC-IB, that models data with a set of Gaussian distributions and that retrieves clusters based on a partial labeling provided by the user (partition-level side information). By combining the ideas from cross-entropy clustering (CEC) with those from the information bottleneck method (IB), our method trades between three conflicting goals: the accuracy with which the data set is modeled, the simplicity of the model, and the consistency of the clustering with side information. Experiments demonstrate that CEC-IB has a performance com…

2017-05-03cs.LGstat.ML

Maximum Entropy Diverse Exploration: Disentangling Maximum Entropy Reinforcement Learning

Binghamton University（宾汉姆顿大学）；Yantai University（烟台大学）

Two hitherto disconnected threads of research, diverse exploration (DE) and maximum entropy RL have addressed a wide range of problems facing reinforcement learning algorithms via ostensibly distinct mechanisms. In this work, we identify a connection between these two approaches. First, a discriminator-based diversity objective is put forward and connected to commonly used divergence measures. We then extend this objective to the maximum entropy framework and propose an algorithm Maximum Entropy Diverse Exploration (MEDE) which provides a principled method to learn diverse behaviors. A theoret…

2019-11-03cs.AIcs.LGstat.ML

Boltzmann Machine Learning with the Latent Maximum Entropy Principle

University of Toronto（多伦多大学）；University of Waterloo（滑铁卢大学）；University of Missouri（密苏里大学）

We present a new statistical learning paradigm for Boltzmann machines based on a new inference principle we have proposed: the latent maximum entropy principle (LME). LME is different both from Jaynes maximum entropy principle and from standard maximum likelihood estimation.We demonstrate the LME principle BY deriving new algorithms for Boltzmann machine parameter estimation, and show how robust and fast new variant of the EM algorithm can be developed.Our experiments show that estimation based on LME generally yields better results than maximum likelihood estimation, particularly when inferri…

2012-10-19cs.LGstat.ML

更多相关论文收起更多论文

最大熵半监督逆强化学习

Maximum Entropy Semi-Supervised Inverse Reinforcement Learning

CMLA, ENS Cachan（CMLA，ENS巴黎-坎帕尼昂）；SequeL team, INRIA Lille - Nord Europe（SequeL团队，INRIA里尔-北欧）；Adobe Research（Adobe研究）

本文提出MESSI算法，结合最大熵逆强化学习与半监督学习原理，利用未标注数据提升性能，在高速驾驶和网格世界任务中验证了其有效性。

2026-10-06cs.LGstat.ML

Generalized Maximum Causal Entropy for Inverse Reinforcement Learning

Singapore Management University（新加坡管理大学）；Nanyang Technological University（南洋理工大学）；Massachusetts Institute of Technology（麻省理工学院）

We consider the problem of learning from demonstrated trajectories with inverse reinforcement learning (IRL). Motivated by a limitation of the classical maximum entropy model in capturing the structure of the network of states, we propose an IRL model based on a generalized version of the causal entropy maximization problem, which allows us to generate a class of maximum entropy IRL models. Our generalized model has an advantage of being able to recover, in addition to a reward function, another expert's function that would (partially) capture the impact of the connecting structure of the stat…

2020-08-18cs.LGstat.ML

Supervised Learning of Labeled Pointcloud Differences via Cover-Tree Entropy Reduction

University of Wisconsin-Stout（威斯康星大学斯托特分校）；Geometric Data Analytics, Inc.（几何数据分析公司）；Duke University（杜克大学）；Columbia University（哥伦比亚大学）

We introduce a new algorithm, called CDER, for supervised machine learning that merges the multi-scale geometric properties of Cover Trees with the information-theoretic properties of entropy. CDER applies to a training set of labeled pointclouds embedded in a common Euclidean space. If typical pointclouds corresponding to distinct labels tend to differ at any scale in any sub-region, CDER can identify these differences in (typically) linear time, creating a set of distributional coordinates which act as a feature extraction mechanism for supervised learning. We describe theoretical properties…

2018-01-19cs.CVcs.LGstat.ML

Approximate maximum entropy principles via Goemans-Williamson with applications to provable variational methods

Princeton University（普林斯顿大学）

The well known maximum-entropy principle due to Jaynes, which states that given mean parameters, the maximum entropy distribution matching them is in an exponential family, has been very popular in machine learning due to its "Occam's razor" interpretation. Unfortunately, calculating the potentials in the maximum-entropy distribution is intractable \cite{bresler2014hardness}. We provide computationally efficient versions of this principle when the mean parameters are pairwise moments: we design distributions that approximately match given pairwise moments, while having entropy which is compara…

2016-07-12cs.DScs.LGstat.ML

Min-max Entropy for Weakly Supervised Pointwise Localization

École de technologie supérieure, Université du Québec（魁北克大学高等技术学院）；Rosalind and Morris Goodman Cancer Research Centre（罗莎琳德与莫里斯·古德曼癌症研究中心）；McGill University（麦吉尔大学）

Pointwise localization allows more precise localization and accurate interpretability, compared to bounding box, in applications where objects are highly unstructured such as in medical domain. In this work, we focus on weakly supervised localization (WSL) where a model is trained to classify an image and localize regions of interest at pixel-level using only global image annotation. Typical convolutional attentions maps are prune to high false positive regions. To alleviate this issue, we propose a new deep learning method for WSL, composed of a localizer and a classifier, where the localizer…

2021-01-21cs.CVcs.LGstat.ML

Maximum Entropy Flow Networks

Maximum entropy modeling is a flexible and popular framework for formulating statistical models given partial knowledge. In this paper, rather than the traditional method of optimizing over the continuous density directly, we learn a smooth and invertible transformation that maps a simple distribution to the desired maximum entropy distribution. Doing so is nontrivial in that the objective being maximized (entropy) is a function of the density itself. By exploiting recent developments in normalizing flow networks, we cast the maximum entropy problem into a finite-dimensional constrained optimi…

2017-04-28stat.MEstat.ML

MedLDA: A General Framework of Maximum Margin Supervised Topic Models

Supervised topic models utilize document's side information for discovering predictive low dimensional representations of documents. Existing models apply the likelihood-based estimation. In this paper, we present a general framework of max-margin supervised topic models for both continuous and categorical response variables. Our approach, the maximum entropy discrimination latent Dirichlet allocation (MedLDA), utilizes the max-margin principle to train supervised topic models and estimate predictive topic representations that are arguably more suitable for prediction tasks. The general princi…

2009-12-30stat.MEstat.ML

通过有效的熵最大化改进预训练自监督嵌入

Improving Pre-trained Self-Supervised Embeddings Through Effective Entropy Maximization

UMass Amherst（马萨诸塞大学阿默斯特分校）；New York University（纽约大学）；Meta - FAIR（元公司FAIR研究院）

针对自监督嵌入熵最大化依赖高维熵估计的问题，本文提出基于易估计低维约束的E2MC附加准则，仅需少量epoch继续训练即可稳定提升下游性能。

2025-03-13cs.CVcs.ITcs.LG