arXiv 2201.091912022-01cs.LG

Revisiting Global Pooling through the Lens of Optimal Transport

Gaoling School of Artificial Intelligence（高瓴人工智能学院）；Renmin University of China（中国人民大学）

Minjie Cheng, Hongteng Xu

英文摘要：

Global pooling is one of the most significant operations in many machine learning models and tasks, whose implementation, however, is often empirical in practice. In this study, we develop a novel and solid global pooling framework through the lens of optimal transport. We demonstrate that most existing global pooling methods are equivalent to solving some specializations of an unbalanced optimal transport (UOT) problem. Making the parameters of the UOT problem learnable, we unify various global pooling methods in the same framework, and accordingly, propose a generalized global pooling layer called UOT-Pooling (UOTP) for neural networks. Besides implementing the UOTP layer based on the classic Sinkhorn-scaling algorithm, we design a new model architecture based on the Bregman ADMM algorithm, which has better numerical stability and can reproduce existing pooling layers more effectively. We test our UOTP layers in several application scenarios, including multi-instance learning, graph classification, and image classification. Our UOTP layers can either imitate conventional global pooling layers or learn some new pooling mechanisms leading to better performance.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Regularized Optimal Transport Layers for Generalized Global Pooling Operations

Renmin University of China（中国人民大学）

Global pooling is one of the most significant operations in many machine learning models and tasks, which works for information fusion and structured data (like sets and graphs) representation. However, without solid mathematical fundamentals, its practical implementations often depend on empirical mechanisms and thus lead to sub-optimal, even unsatisfactory performance. In this work, we develop a novel and generalized global pooling framework through the lens of optimal transport. The proposed framework is interpretable from the perspective of expectation-maximization. Essentially, it aims at…

2022-12-13cs.CVcs.LGstat.ML

Bayesian Coresets: Revisiting the Nonconvex Optimization Perspective

University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, Berkeley（加州大学伯克利分校）；Rice University（莱斯大学）

Bayesian coresets have emerged as a promising approach for implementing scalable Bayesian inference. The Bayesian coreset problem involves selecting a (weighted) subset of the data samples, such that the posterior inference using the selected subset closely approximates the posterior inference using the full dataset. This manuscript revisits Bayesian coresets through the lens of sparsity constrained optimization. Leveraging recent advances in accelerated optimization methods, we propose and analyze a novel algorithm for coreset selection. We provide explicit convergence rate guarantees and pre…

2021-02-25cs.LGstat.COstat.ML

Approximation and Learning with Deep Convolutional Models: a Kernel Perspective

New York University（纽约大学）

The empirical success of deep convolutional networks on tasks involving high-dimensional data such as images or audio suggests that they can efficiently approximate certain functions that are well-suited for such tasks. In this paper, we study this through the lens of kernel methods, by considering simple hierarchical kernels with two or three convolution and pooling layers, inspired by convolutional kernel networks. These achieve good empirical performance on standard vision datasets, while providing a precise description of their functional space that yields new insights on their inductive b…

2022-03-18cs.LGstat.ML

通过参数冗余视角重新审视LoRA：谱编码有助于提升性能

Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps

The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））

本文重新审视LoRA中的参数冗余，提出SeLoRA，利用谱编码从稀疏谱子空间重新参数化LoRA，以更少参数提升效率，并在多种下游任务上超越强基线。

2025-06-20cs.LG

Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Framework

MIT（麻省理工学院）；MSU（密歇根州立大学）；IBM Research AI（IBM研究院人工智能中心）；UCSD（加州大学圣地亚哥分校）

As a seminal tool in self-supervised representation learning, contrastive learning has gained unprecedented attention in recent years. In essence, contrastive learning aims to leverage pairs of positive and negative samples for representation learning, which relates to exploiting neighborhood information in a feature space. By investigating the connection between contrastive learning and neighborhood component analysis (NCA), we provide a novel stochastic nearest neighbor viewpoint of contrastive learning and subsequently propose a series of contrastive losses that outperform the existing ones…

2022-01-28cs.CVcs.LG

重新审视零阶Hessian近似：单步策略优化视角

Revisiting Zeroth-Order Hessian Approximation: A Single-Step Policy Optimization Lens

Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Shanghai University of Finance and Economics（上海财经大学）

本文通过单步策略优化视角统一零阶Hessian估计，提出方差缩减的ZoVH框架，实现全Hessian矩阵、正则化逆及偏差校正逆Hessian-梯度积的高效估计。

2026-05-29cs.LG

更多相关论文收起更多论文

Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimization

Michigan State University（密歇根州立大学）；UC Santa Barbara（加州大学圣巴巴拉分校）；University of Minnesota（明尼苏达大学）；MIT-IBM Watson AI Lab, IBM Research（IBM研究院麻省理工-IBM沃森人工智能实验室）

Adversarial training (AT) is a widely recognized defense mechanism to gain the robustness of deep neural networks against adversarial attacks. It is built on min-max optimization (MMO), where the minimizer (i.e., defender) seeks a robust model to minimize the worst-case training loss in the presence of adversarial examples crafted by the maximizer (i.e., attacker). However, the conventional MMO method makes AT hard to scale. Thus, Fast-AT (Wong et al., 2020) and other recent algorithms attempt to simplify MMO by replacing its maximization step with the single gradient sign-based attack generat…

2022-10-04cs.LG

从黎曼几何视角理解协方差池化中的矩阵函数归一化

Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometry

University of Trento（特伦托大学）；Caltech（加州理工学院）；Jiangnan University（江南大学）；Cisco Systems（思科系统公司）

本文从黎曼几何视角统一解释了全局协方差池化中矩阵对数与矩阵幂归一化的作用机制，指出其有效性源于隐含的黎曼分类器，并通过细粒度与大规模视觉分类实验验证了该结论。

2025-02-07cs.CVcs.LG

通过不可约表示的视角重新审视多置换等变性

Revisiting Multi-Permutation Equivariance through the Lens of Irreducible Representations

Technion – Israel Institute of Technology（以色列理工学院）

本文基于不可约表示与舒尔引理提出新方法，推导了DeepSets等模型，扩展到未对齐对称集时发现大量非暹罗式层，经实验其可提升多项任务性能。

2026-08-25cs.AIcs.LG

Transolver 是一个线性变换器：通过线性注意力重新审视物理注意力

Transolver is a Linear Transformer: Revisiting Physics-Attention through the Lens of Linear Attention

本文提出线性注意力神经算子（LinearNO），通过将物理注意力重新表述为线性注意力，提升PDE求解效率，实现参数和计算成本的显著降低。

2026-01-07cs.AIcs.LG

从低熵与动态稀疏性的视角重新审视Transformers

Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity

Gaoling School of Artificial Intelligence（人工智能学院）；Renmin University of China（中国人民大学）

本文从低熵和动态稀疏性视角研究Transformers，发现模型倾向于学习更低熵分布且FFN模块起关键作用，同时大模型更倾向绕过attention计算并伴随死亡神经元突增导致的训练不稳定。

2025-04-26cs.AIcs.LG

Revisiting local branching with a machine learning lens

Polytechnique Montréal（蒙特利尔理工学院）；University of Padua（帕多瓦大学）；Jacobs Technion-Cornell Institute（雅各布斯特克尼恩-康奈尔研究所）；Cornell Tech（康奈尔科技校区）；Technion - IIT（以色列理工学院）

Finding high-quality solutions to mixed-integer linear programming problems (MILPs) is of great importance for many practical applications. In this respect, the refinement heuristic local branching (LB) has been proposed to produce improving solutions and has been highly influential for the development of local search methods in MILP. The algorithm iteratively explores a sequence of solution neighborhoods defined by the so-called local branching constraint, namely, a linear inequality limiting the distance from a reference solution. For a LB algorithm, the choice of the neighborhood size is cr…

2022-08-03cs.AIcs.DMcs.LG

通过Gromov-Wasserstein信息瓶颈的视角重新审视反事实回归

Revisiting Counterfactual Regression through the Lens of Gromov-Wasserstein Information Bottleneck

Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）

本研究提出Gromov-Wasserstein信息瓶颈（GWIB）范式，通过最大化潜在表示与结果的互信息并惩罚与协变量的核化互信息，以抑制选择偏差，从而提升反事实回归在个体化治疗效果估计中的性能。

2024-05-24cs.AIcs.LGstat.ML

Deep Graph Mapper: Seeing Graphs through the Neural Lens

University of Cambridge（剑桥大学）

Recent advancements in graph representation learning have led to the emergence of condensed encodings that capture the main properties of a graph. However, even though these abstract representations are powerful for downstream tasks, they are not equally suitable for visualisation purposes. In this work, we merge Mapper, an algorithm from the field of Topological Data Analysis (TDA), with the expressive power of Graph Neural Networks (GNNs) to produce hierarchical, topologically-grounded visualisations of graphs. These visualisations do not only help discern the structure of complex graphs but…

2020-02-20cs.LGcs.SIstat.ML

关于具有等变性、局部性和权重共享的单隐层网络的样本复杂度

On the Sample Complexity of One Hidden Layer Networks with Equivariance, Locality and Weight Sharing

Qualcomm AI Research（高通人工智能研究部门）；Qualcomm Technologies, Inc.（高通技术公司）

本文通过统计学习理论，分析了单隐层网络中权重共享、等变性和局部性对样本复杂度的影响，给出了维度无关的界，并指出局部性与表达力存在权衡。

2025-01-23cs.LGmath.STstat.ML

重新审视通过因果能量最小化进行Transformer层参数化

Revisiting Transformer Layer Parameterization Through Causal Energy Minimization

Microsoft（微软）；Microsoft Research Cambridge（微软研究院剑桥）

本文提出因果能量最小化框架，将Transformer层视为条件能量函数的优化步骤，揭示了权重共享、低秩交互等设计空间，验证了在百万人参数规模下，能量引导的层设计能稳定训练并匹配基线模型。

2026-05-08cs.AIcs.LGstat.ML