We present L0Learn: an open-source package for sparse linear regression and classification using $\ell_0$ regularization. L0Learn implements scalable, approximate algorithms, based on coordinate descent and local combinatorial optimization. The package is built using C++ and has user-friendly R and Python interfaces. L0Learn can address problems with millions of features, achieving competitive run times and statistical performance with state-of-the-art sparse learning packages. L0Learn is available on both CRAN and GitHub (https://cran.r-project.org/package=L0Learn and https://github.com/hazimehh/L0Learn).
In this paper, we present a new R package COREclust dedicated to the detection of representative variables in high dimensional spaces with a potentially limited number of observations. Variable sets detection is based on an original graph clustering strategy denoted CORE-clustering algorithm that detects CORE-clusters, i.e. variable sets having a user defined size range and in which each variable is very similar to at least another variable. Representative variables are then robustely estimate as the CORE-cluster centers. This strategy is entirely coded in C++ and wrapped by R using the Rcpp p…
BHAM is a freely avaible R pakcage that implments Bayesian hierarchical additive models for high-dimensional clinical and genomic data. The package includes functions that generalized additive model, and Cox additive model with the spike-and-slab LASSO prior. These functions implement scalable and stable algorithms to estimate parameters. BHAM also provides utility functions to construct additive models in high dimensional settings, select optimal models, summarize bi-level variable selection results, and visualize nonlinear effects. The package can facilitate flexible modeling of large-scale …
Advancements in medical informatics tools and high-throughput biological experimentation make large-scale biomedical data routinely accessible to researchers. Competing risks data are typical in biomedical studies where individuals are at risk to more than one cause (type of event) which can preclude the others from happening. The Fine-Gray model is a popular and well-appreciated model for competing risks data and is currently implemented in a number of statistical software packages. However, current implementations are not computationally scalable for large-scale competing risks data. We have…
Regularized Ordinal Regression and the ordinalNet R Package
正则化技术，如套索（Tibshirani 1996）和弹性网（Zou and Hastie 2005），可用于提高回归模型系数估计和预测精度，以及执行变量选择。有序回归模型被广泛应用于正则化的应用中;然而，这些模型并不包含在正则化回归的许多流行软件包中。我们提出了一个坐标下降算法，以适应广泛的一类有序回归模型与弹性网罚款。此外，我们证明了这类模型中的每个模型都可以推广到更灵活的形式，例如可以容纳无序的分类数据。我们引入了一个适用于这两种模型形式的弹性网惩罚类。此外，该惩罚可用于将非有序模型向其有序对应物收缩。最后，我们介绍了R包ordinalNet，它实现了这个模型类的算法。
Pre-compiled binary packages provide a convenient way of efficiently distributing software that has been adopted by most Linux package management systems. However, the heterogeneity of the Linux ecosystem, combined with the growing number of R extensions available, poses a scalability problem. As a result, efforts to bring binary R packages to Linux have been scattered, and lack a proper mechanism to fully integrate them with R's package manager. This work reviews past and present of binary distribution for Linux, and presents a path forward by showcasing the `cran2copr' project, an RPM-based …
Multiple product attributes like dimensions, weight, fragility, liquid content etc. determine the package type used by e-commerce companies to ship products. Sub-optimal package types lead to damaged shipments, incurring huge damage related costs and adversely impacting the company's reputation for safe delivery. Items can be shipped in more protective packages to reduce damage costs, however this increases the shipment costs due to expensive packaging and higher transportation costs. In this work, we propose a multi-stage approach that trades-off between shipment and damage costs for each pro…
Unit root tests form an essential part of any time series analysis. We provide practitioners with a single, unified framework for comprehensive and reliable unit root testing in the R package bootUR.The package's backbone is the popular augmented Dickey-Fuller test paired with a union of rejections principle, which can be performed directly on single time series or multiple (including panel) time series. Accurate inference is ensured through the use of bootstrap methods. The package addresses the needs of both novice users, by providing user-friendly and easy-to-implement functions with sensib…
This paper presents the R package PlackettLuce, which implements a generalization of the Plackett-Luce model for rankings data. The generalization accommodates both ties (of arbitrary order) and partial rankings (complete rankings of subsets of items). By default, the implementation adds a set of pseudo-comparisons with a hypothetical item, ensuring that the underlying network of wins and losses between items is always strongly connected. In this way, the worth of each item always has a finite maximum likelihood estimate, with finite standard error. The use of pseudo-comparisons also has a reg…
Although a number of studies have developed fast geographically weighted regression (GWR) algorithms for large samples, none of them has achieved linear-time estimation, which is considered a requisite for big data analysis in machine learning, geostatistics, and related domains. Against this backdrop, this study proposes a scalable GWR (ScaGWR) for large datasets. The key improvement is the calibration of the model through a pre-compression of the matrices and vectors whose size depends on the sample size, prior to the leave-one-out cross-validation, which is the heaviest computational step i…
Rutgers University - New Brunswick（罗格斯大学新布朗斯维克分校）；Weifang Medical University（潍坊医学院）
The paper presents Imbalance-XGBoost, a Python package that combines the powerful XGBoost software with weighted and focal losses to tackle binary label-imbalanced classification tasks. Though a small-scale program in terms of size, the package is, to the best of the authors' knowledge, the first of its kind which provides an integrated implementation for the two losses on XGBoost and brings a general-purpose extension on XGBoost for label-imbalanced scenarios. In this paper, the design and usage of the package are described with exemplar code listings, and its convenience to be integrated int…
With the advent of high-throughput sequencing (HTS) in molecular biology and medicine, the need for scalable statistical solutions for modeling complex biological systems has become of critical importance. The increasing number of platforms and possible experimental scenarios raised the problem of integrating large amounts of new heterogeneous data and current knowledge, to test novel hypotheses and improve our comprehension of physiological processes and diseases. Although network theory provided a framework to represent biological systems and study their hidden properties, different algorith…
EarlyStopping: Implicit Regularization for Iterative Learning Procedures in Python
Institute of Mathematics, Humboldt-Universität zu Berlin（柏林洪堡大学数学研究所）；School of Business and Economics, Humboldt-Universität zu Berlin（柏林洪堡大学工商经济学院）；Institute of Mathematics, Universität Potsdam（波茨坦大学数学研究所）
In this paper, we consider a generic probabilistic discriminative learner from the functional viewpoint and argue that, to make it learn well, it is necessary to constrain its hypothesis space to a set of non-trivial piecewise constant functions. To achieve this goal, we present a scalable unsupervised regularization framework. On the theoretical front, we prove that this framework is conducive to a factually confident and smooth discriminative model and connect it to an adversarial Taboo game, spectral clustering and virtual adversarial training. Experimentally, we take deep neural networks a…
We propose a simple and easy to implement neural network compression algorithm that achieves results competitive with more complicated state-of-the-art methods. The key idea is to modify the original optimization problem by adding K independent Gaussian priors (corresponding to the k-means objective) over the network parameters to achieve parameter quantization, as well as an L1 penalty to achieve pruning. Unlike many existing quantization-based methods, our method uses hard clustering assignments of network parameters, which adds minimal change or overhead to standard network training. We als…
University of Minnesota, Twin Cities（明尼苏达大学双城分校）；Max Planck Institute for Dynamics of Complex Technical Systems（马克斯·普朗克复杂技术系统动力学研究所）
Optimizing nonconvex (NCVX) problems, especially nonsmooth and constrained ones, is an essential part of machine learning. However, it can be hard to reliably solve such problems without optimization expertise. Existing general-purpose NCVX optimization packages are powerful but typically cannot handle nonsmoothness. GRANSO is among the first optimization solvers targeting general nonsmooth NCVX problems with nonsmooth constraints, but, as it is implemented in MATLAB and requires the user to provide analytical gradients, GRANSO is often not a convenient choice in machine learning (especially d…
In this paper, we propose a novel graph-based approach for semi-supervised learning problems, which considers an adaptive adjacency of the examples throughout the unsupervised portion of the training. Adjacency of the examples is inferred using the predictions of a neural network model which is first initialized by a supervised pretraining. These predictions are then updated according to a novel unsupervised objective which regularizes another adjacency, now linking the output nodes. Regularizing the adjacency of the output nodes, inferred from the predictions of the network, creates an easier…
University of Colorado Boulder（科罗拉多大学博尔德分校）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Gaussian processes (GPs) are Bayesian non-parametric models popular in a variety of applications due to their accuracy and native uncertainty quantification (UQ). Tuning GP hyperparameters is critical to ensure the validity of prediction accuracy and uncertainty; uniquely estimating multiple hyperparameters in, e.g. the Matern kernel can also be a significant challenge. Moreover, training GPs on large-scale datasets is a highly active area of research: traditional maximum likelihood hyperparameter training requires quadratic memory to form the covariance matrix and has cubic training complexit…
University of Padova（帕多瓦大学）；Samsung AI Research（三星人工智能研究院）
Several techniques have recently aimed to improve the performance of deep learning models for Scene Graph Generation (SGG) by incorporating background knowledge. State-of-the-art techniques can be divided into two families: one where the background knowledge is incorporated into the model in a subsymbolic fashion, and another in which the background knowledge is maintained in symbolic form. Despite promising results, both families of techniques face several shortcomings: the first one requires ad-hoc, more complex neural architectures increasing the training or inference cost; the second one s…
DomainLab: A modular Python package for domain generalization in deep learning
Institute of AI for Health, Helmholtz Munich（慕尼黑亥姆霍兹人工智能健康研究所）；Technical University of Munich（慕尼黑工业大学）；Division of Imaging, Diagnostics, and Software Reliability, CDRH U.S Food and Drug Administration（美国食品药品监督管理局设备与放射卫生中心成像、诊断与软件可靠性部）；Medical University of Innsbruck（因斯布鲁克医科大学）；Machine Learning Research Lab, Volkswagen Group（大众集团机器学习研究实验室）；Data & Analytics (D&A), Roche Pharma Research and Early Development (pRED), Roche Innovation Center Munich（罗氏制药研发与早期开发部数据与分析部门（慕尼黑罗氏创新中心））；Helmholtz AI（亥姆霍兹人工智能）；German Cancer Research Center (DKFZ)（德国癌症研究中心）
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Unlike areas such as computer vision and speech recognition where convolutional and recurrent neural networks-based approaches have proven effective to the nature of the respective areas of application, deep learning (DL) still lacks a general approach suitable for the unique nature and challenges of RF systems such as radar, signals intelligence, electronic warfare, and communications. Existing approaches face problems in robustness, consistency, efficiency, repeatability and scalability. One of the main challenges in RF sensing such as radar target identification is the difficulty and cost o…
In this paper, we introduce Deep Probabilistic Ensembles (DPEs), a scalable technique that uses a regularized ensemble to approximate a deep Bayesian Neural Network (BNN). We do so by incorporating a KL divergence penalty term into the training objective of an ensemble, derived from the evidence lower bound used in variational inference. We evaluate the uncertainty estimates obtained from our models for active learning on visual classification. Our approach steadily improves upon active learning baselines as the annotation budget is increased.
ControlBurn is a Python package to construct feature-sparse tree ensembles that support nonlinear feature selection and interpretable machine learning. The algorithms in this package first build large tree ensembles that prioritize basis functions with few features and then select a feature-sparse subset of these basis functions using a weighted lasso optimization criterion. The package includes visualizations to analyze the features selected by the ensemble and their impact on predictions. Hence ControlBurn offers the accuracy and flexibility of tree-ensemble models and the interpretability o…
Least Squares Revisited: Scalable Approaches for Multi-class Prediction
Microsoft Research（微软研究院）；Microsoft Cloud and Information Services Lab（微软云与信息服务实验室）；College of Computing, Georgia Tech（佐治亚理工学院计算学院）；Stanford University（斯坦福大学）
Carnegie Mellon University（卡内基梅隆大学）；University of Toronto（多伦多大学）；Northeastern University Toronto（东北大学多伦多校区）
PyOD is an open-source Python toolbox for performing scalable outlier detection on multivariate data. Uniquely, it provides access to a wide range of outlier detection algorithms, including established outlier ensembles and more recent neural network-based approaches, under a single, well-documented API designed for use by both practitioners and researchers. With robustness and scalability in mind, best practices such as unit testing, continuous integration, code coverage, maintainability checks, interactive examples and parallelization are emphasized as core components in the toolbox's develo…
