This paper presents a model-agnostic ensemble approach for supervised learning. The proposed approach is based on a parametric version of Random Subspace, in which each base model is learned from a feature subset sampled according to a Bernoulli distribution. Parameter optimization is performed using gradient descent and is rendered tractable by using an importance sampling approach that circumvents frequent re-training of the base models after each gradient descent step. The degree of randomization in our parametric Random Subspace is thus automatically tuned through the optimization of the feature selection probabilities. This is an advantage over the standard Random Subspace approach, where the degree of randomization is controlled by a hyper-parameter. Furthermore, the optimized feature selection probabilities can be interpreted as feature importance scores. Our algorithm can also easily incorporate any differentiable regularization term to impose constraints on these importance scores.
Automated Random Embedding for Practical Bayesian Optimization with Unknown Effective Dimension
Shanghai Institute of AI for Education, and School of Computer Science and Technology, East China Normal University（上海人工智能教育研究院，东华大学计算机科学与技术学院）；Ant Group（蚂蚁集团）；Nanjing University（南京大学）
Northeastern University（东北大学）；IBM Research – Almaden（IBM阿尔马登研究院）；Google Research, Brain Team（谷歌研究院大脑团队）
Private data analysis suffers a costly curse of dimensionality. However, the data often has an underlying low-dimensional structure. For example, when optimizing via gradient descent, the gradients often lie in or near a low-dimensional subspace. If that low-dimensional structure can be identified, then we can avoid paying (in terms of privacy or accuracy) for the high ambient dimension. We present differentially private algorithms that take input data sampled from a low-dimensional linear subspace (possibly with a small amount of error) and output that subspace (or an approximation to it). Th…
University of Iowa（爱荷华大学）；Tippie College of Business, University of Iowa（爱荷华大学蒂皮商学院）
Driven by an increasing need for model interpretability, interpretable models have become strong competitors for black-box models in many real applications. In this paper, we propose a novel type of model where interpretable models compete and collaborate with black-box models. We present the Model-Agnostic Linear Competitors (MALC) for partially interpretable classification. MALC is a hybrid model that uses linear models to locally substitute any black-box model, capturing subspaces that are most likely to be in a class while leaving the rest of the data to the black-box. MALC brings together…
Learning 3D geometry directly from raw data, such as point clouds, triangle soups, or unoriented meshes is still a challenging task that feeds many downstream computer vision and graphics applications. In this paper, we introduce SALD: a method for learning implicit neural representations of shapes directly from raw data. We generalize sign agnostic learning (SAL) to include derivatives: given an unsigned distance function to the input raw data, we advocate a novel sign agnostic regression loss, incorporating both pointwise values and gradients of the unsigned distance function. Optimizing thi…
Fudan University（复旦大学）；University of Texas at Dallas（德克萨斯大学达拉斯分校）
A large body of literature has proved that the Bayesian optimization framework is especially efficient and effective in analog circuit synthesis. However, most of the previous research works only focus on designing informative surrogate models or efficient acquisition functions. Even if searching for the global optimum over the acquisition function surface is itself a difficult task, it has been largely ignored. In this paper, we propose a fast and robust Bayesian optimization approach via one-dimensional subspaces for analog circuit synthesis. By solely focusing on optimizing one-dimension su…
The Generative Leap: Sharp Sample Complexity for Efficiently Learning Gaussian Multi-Index Models
PACM, Princeton University（普林斯顿大学）；Electrical and Computer Engineering Department, Princeton University（普林斯顿大学电子与计算机工程系）；Courant Institute of Mathematical Sciences, New York University Center for Data Science, New York University（纽约大学应用数学科学学院，纽约大学数据科学中心）
本文针对高斯多索引模型提出生成性跃迁指数，证明低次多项式算法下样本复杂度为 Θ(d^{1∨k/2})，并给出基于 Hermite 张量谱 U 统计量的不可知估计方法达到该下界。
Although many machine learning algorithms involve learning subspaces with particular characteristics, optimizing a parameter matrix that is constrained to represent a subspace can be challenging. One solution is to use Riemannian optimization methods that enforce such constraints implicitly, leveraging the fact that the feasible parameter values form a manifold. While Riemannian methods exist for some specific problems, such as learning a single subspace, there are more general subspace constraints that offer additional flexibility when setting up an optimization problem, but have not been for…
IBM Almaden Research Center（IBM阿尔姆登研究中心）；Carnegie Mellon University（卡内基梅隆大学）；College of Computing, Georgia Institute of Technology（佐治亚理工学院计算学院）
We prove the following strong hardness result for learning: Given a distribution of labeled examples from the hypercube such that there exists a monomial consistent with $(1-\eps)$ of the examples, it is NP-hard to find a halfspace that is correct on $(1/2+\eps)$ of the examples, for arbitrary constants $\eps > 0$. In learning theory terms, weak agnostic learning of monomials is hard, even if one is allowed to output a hypothesis from the much bigger concept class of halfspaces. This hardness result subsumes a long line of previous results, including two recent hardness results for the proper …
The Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）
Whilst adversarial attack detection has received considerable attention, it remains a fundamentally challenging problem from two perspectives. First, while threat models can be well-defined, attacker strategies may still vary widely within those constraints. Therefore, detection should be considered as an open-set problem, standing in contrast to most current detection approaches. These methods take a closed-set view and train binary detectors, thus biasing detection toward attacks seen during detector training. Second, limited information is available at test time and typically confounded by …
National Institute of Standards and Technology（美国国家标准与技术研究院）
To meet the ever-growing demands of data throughput for forthcoming and deployed wireless networks, new wireless technologies like Long-Term Evolution License-Assisted Access (LTE-LAA) operate in shared and unlicensed bands. However, the LAA network must co-exist with incumbent IEEE 802.11 Wi-Fi systems. We consider a coexistence scenario where multiple LAA and Wi-Fi links share an unlicensed band. We aim to improve this coexistence by maximizing the key performance indicators (KPIs) of these networks simultaneously via dimension reduction and multi-criteria optimization. These KPIs are networ…
Stochastic Gradient Descent (SGD) has proven to be remarkably effective in optimizing deep neural networks that employ ever-larger numbers of parameters. Yet, improving the efficiency of large-scale optimization remains a vital and highly active area of research. Recent work has shown that deep neural networks can be optimized in randomly-projected subspaces of much smaller dimensionality than their native parameter space. While such training is promising for more efficient and scalable optimization schemes, its practical application is limited by inferior optimization performance. Here, we im…
Bayesian optimization is a popular method for optimizing expensive black-box functions. Yet it oftentimes struggles in high dimensions where the computation could be prohibitively heavy. To alleviate this problem, we introduce Coordinate backoff Bayesian Optimization (CobBO) with two-stage kernels. During each round, the first stage uses a simple coarse kernel that sacrifices the approximation accuracy for computational efficiency. It captures the global landscape by purposely smoothing away local fluctuations. Then, in the second stage of the same round, past observed points in the full space…
Optimizing Polynomial Graph Filters: A Novel Adaptive Krylov Subspace Approach
National University of Singapore（新加坡国立大学）；OPPO Shenzhen Research Institute（OPPO深圳研究院）；Micro Connect Technology, China（中国微联通技术公司）；University of Cambridge（剑桥大学）
Metric-agnostic Learning-to-Rank via Boosting and Rank Approximation
Dept. of Statistics \& Data Science University of Central Florida Orlando, USA；Department of CIS, SKL-IOTSC University of Macau Macao, China；School of Computing；AI Arizona State University Tempe, USA
Recent Self-Supervised Learning (SSL) methods are able to learn feature representations that are invariant to different data augmentations, which can then be transferred to downstream tasks of interest. However, different downstream tasks require different invariances for their best performance, so the optimal choice of augmentations for SSL depends on the target task. In this paper, we aim to learn self-supervised features that generalize well across a variety of downstream tasks (e.g., object classification, detection and instance segmentation) without knowing any task information beforehand…
High-dimensional spatio-temporal dynamics can often be encoded in a low-dimensional subspace. Engineering applications for modeling, characterization, design, and control of such large-scale systems often rely on dimensionality reduction to make solutions computationally tractable in real-time. Common existing paradigms for dimensionality reduction include linear methods, such as the singular value decomposition (SVD), and nonlinear methods, such as variants of convolutional autoencoders (CAE). However, these encoding techniques lack the ability to efficiently represent the complexity associat…
Aalto University（阿尔托大学）；University of British Columbia（不列颠哥伦比亚大学）；Stanford University（斯坦福大学）
We propose a novel method for exploring the dynamics of physically based animated characters, and learning a task-agnostic action space that makes movement optimization easier. Like several previous papers, we parameterize actions as target states, and learn a short-horizon goal-conditioned low-level control policy that drives the agent's state towards the targets. Our novel contribution is that with our exploration data, we are able to learn the low-level policy in a generic manner and without any reference movement data. Trained once for each agent or simulation environment, the policy impro…
Few-shot Multi-Task Learning of Linear Invariant Features with Meta Subspace Pursuit
School of Mathematical Sciences, Shanghai Jiao Tong University（上海交通大学数学科学学院）；Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University（上海交通大学自然科学研究院）；SJTU-Yale Joint Center for Biostatistics and Data Science, Shanghai Jiao Tong University（上海交通大学-耶鲁大学生物统计与数据科学联合中心）
Google Brain Robotics（谷歌大脑机器人部门）；UC Berkeley（加州大学伯克利分校）；Columbia University（哥伦比亚大学）
We present a new algorithm ASEBO for optimizing high-dimensional blackbox functions. ASEBO adapts to the geometry of the function and learns optimal sets of sensing directions, which are used to probe it, on-the-fly. It addresses the exploration-exploitation trade-off of blackbox optimization with expensive blackbox queries by continuously learning the bias of the lower-dimensional model used to approximate gradients of smoothings of the function via compressed sensing and contextual bandits methods. To obtain this model, it leverages techniques from the emerging theory of active subspaces in …
CDS4RAG: Cyclic Dual-Sequential Hyperparameter Optimization for RAG
School of Computer Science and Engineering, UESTC, Chengdu, China（电子科技大学计算机科学与工程学院，成都，中国）；IDEAS Lab, University of Birmingham, Birmingham, UK（伯明翰大学IDEAS实验室，英国布里斯托尔）
