In this paper, we first introduce batch normalization to the edRVFL network. This re-normalization method can help the network avoid divergence of the hidden features. Then we propose novel variants of Ensemble Deep Random Vector Functional Link (edRVFL). Weighted edRVFL (WedRVFL) uses weighting methods to give training samples different weights in different layers according to how the samples were classified confidently in the previous layer thereby increasing the ensemble's diversity and accuracy. Furthermore, a pruning-based edRVFL (PedRVFL) has also been proposed. We prune some inferior neurons based on their importance for classification before generating the next hidden layer. Through this method, we ensure that the randomly generated inferior features will not propagate to deeper layers. Subsequently, the combination of weighting and pruning, called Weighting and Pruning based Ensemble Deep Random Vector Functional Link Network (WPedRVFL), is proposed. We compare their performances with other state-of-the-art deep feedforward neural networks (FNNs) on 24 tabular UCI classification datasets. The experimental results illustrate the superior performance of our proposed methods.
Eltwise layer is a commonly used structure in the multi-branch deep learning network. In a filter-wise pruning procedure, due to the specific operation of the eltwise layer, all its previous convolutional layers should vote for which filters by index should be pruned. Since only an intersection of the voted filters is pruned, the compression rate is limited. This work proposes a method called Directed-Weighting Group Lasso (DWGL), which enforces an index-wise incremental (directed) coefficient on the filterlevel group lasso items, so that the low index filters getting high activation tend to b…
Portland State University（波特兰州立大学）；Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心）；University of Chinese Academy of Sciences（中国科学院大学）；Dalian Maritime University（大连海事大学）；Jilin University（吉林大学）
Back to Fundamentals: Low-Level Visual Features Guided Progressive Token Pruning
Xidian University（西安电子科技大学）；University of Michigan – Shanghai Jiao Tong University Joint Institute（密歇根大学-上海交通大学联合学院）；Shanghai Jiao Tong University（上海交通大学）；Norwegian University of Science and Technology（挪威科技大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）
How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle
Zhejiang University（浙江大学）；Westlake University（西湖大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Carnegie Mellon University（卡内基梅隆大学）
The University of Chicago（芝加哥大学）；Google Research（谷歌研究院）
We study the current best model (KDG) for question answering on tabular data evaluated over the WikiTableQuestions dataset. Previous ablation studies performed against this model attributed the model's performance to certain aspects of its architecture. In this paper, we find that the model's performance also crucially depends on a certain pruning of the data used to train the model. Disabling the pruning step drops the accuracy of the model from 43.3% to 36.3%. The large impact on the performance of the KDG model suggests that the pruning may be a useful pre-processing step in training other …
Weight-Informed Self-Explaining Clustering for Mixed-Type Tabular Data
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）智能科学与工程学院）
Tel Aviv University（特拉维夫大学）；Playtika Ltd.（Playtika公司）；Bar-Ilan University（巴伊兰大学）
When presented with a binary classification problem where the data exhibits severe class imbalance, most standard predictive methods may fail to accurately model the minority class. We present a model based on Generative Adversarial Networks which uses additional regularization losses to map majority samples to corresponding synthetic minority samples. This translation mechanism encourages the synthesized samples to be close to the class boundary. Furthermore, we explore a selection criterion to retain the most useful of the synthesized samples. Experimental results using several downstream cl…
A Systematic Evaluation of Imbalance Handling Methods in Biomedical Binary Classification
Institute for Health Informatics, University of Minnesota（明尼苏达大学健康信息学研究所）；Department of Computer Science and Engineering, University of Minnesota（明尼苏达大学计算机科学与工程系）；School of Statistics, University of Minnesota（明尼苏达大学统计学系）；Division of Computational Health Sciences, Department of Surgery, University of Minnesota（明尼苏达大学外科系计算健康科学分会）
We present BRIDGE, a powerful sequential architecture for modeling dependencies between natural language questions and relational databases in cross-DB semantic parsing. BRIDGE represents the question and DB schema in a tagged sequence where a subset of the fields are augmented with cell values mentioned in the question. The hybrid sequence is encoded by BERT with minimal subsequent layers and the text-DB contextualization is realized via the fine-tuned deep attention in BERT. Combined with a pointer-generator decoder with schema-consistency driven search space pruning, BRIDGE attained state-o…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Colby College（科尔比学院）
Gradient boosted decision trees are some of the most popular algorithms in applied machine learning. They are a flexible and powerful tool that can robustly fit to any tabular dataset in a scalable and computationally efficient way. One of the most critical parameters to tune when fitting these models are the various penalty terms used to distinguish signal from noise in the current model. These penalties are effective in practice, but are lacking in robust theoretical justifications. In this paper we develop and present a novel theoretically justified hypothesis test of split quality for grad…
Free Lunch in the Forest: Functionally-Identical Pruning of Boosted Tree Ensembles
Polytechnique Montréal（蒙特利尔理工学院）；Centre Interuniversitaire de Recherche sur les Réseaux d’Entreprise, la Logistique et le Transport (CIRRELT)（企业网络、物流与运输跨大学研究中心（CIRRELT））
Fed-TGAN: Federated Learning Framework for Synthesizing Tabular Data
TU Delft（代尔夫特理工大学）；ABB Corporate Research Switzerland（瑞士ABB企业研究）；Institute for Clarity in Documentation（文档清晰研究所）；Inria Paris-Rocquencourt（巴黎-罗克琴堡研究所）；Rajiv Gandhi University（拉贾·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒尔研究实验室）
Department of Biostatistics, University of Michigan（密歇根大学生物统计学系）；Departments of Biomedical Data Science and Statistics, Stanford University（斯坦福大学生物医学数据科学与统计学系）
Enhancing Tabular Data Optimization with a Flexible Graph-based Reinforced Exploration Strategy
CNIC, CAS（中国科学院计算机网络信息中心）；University of CAS（中国科学院大学）；University of Kansas（堪萨斯大学）；Great Bay University（大湾区大学）；IR A*STAR（新加坡科技研究局智能研究院）；Tsinghua University（清华大学）
Predicting Live Birth After Fresh Embryo Transfer in IVF/ICSI: Differential ICM/Trophectoderm Morphology Encoding and a Hybrid Attention--Logistic Regression Ensemble
School of Computer Science, Wuhan University（武汉大学计算机学院）；Zhongnan Hospital of Wuhan University（武汉大学中南医院）；Clinical Medicine Research Center of Prenatal Diagnosis and Birth Health in Hubei Province（湖北省产前诊断与出生健康临床医学研究中心）
Recursive Learning Without Collapse: A Weighting-Based Stabilization Framework
Wang Yanan Institute for Studies in Economics, Xiamen University（厦门大学王亚南经济研究所）；Department of Statistics and Data Science, University of California, Los Angeles（加州大学洛杉矶分校统计与数据科学系）
QMIX is a popular $Q$-learning algorithm for cooperative MARL in the centralised training and decentralised execution paradigm. In order to enable easy decentralisation, QMIX restricts the joint action $Q$-values it can represent to be a monotonic mixing of each agent's utilities. However, this restriction prevents it from representing value functions in which an agent's ordering over its actions can depend on other agents' actions. To analyse this representational limitation, we first formalise the objective QMIX optimises, which allows us to view QMIX as an operator that first computes the $…
École Polytechnique（巴黎综合理工学院）；CNRS（法国国家科学研究中心）；Université de Paris（巴黎大学）；Ecole normale supérieure（巴黎高等师范学院）
Random Forests (RF) is one of the algorithms of choice in many supervised learning applications, be it classification or regression. The appeal of such tree-ensemble methods comes from a combination of several characteristics: a remarkable accuracy in a variety of tasks, a small number of parameters to tune, robustness with respect to features scaling, a reasonable computational cost for training and prediction, and their suitability in high-dimensional settings. The most commonly used RF variants however are "offline" algorithms, which require the availability of the whole dataset at once. In…
Energy Systems Institute SB RAS（俄罗斯科学院西伯利亚分院能源系统研究所）；Irkutsk State University（伊尔库茨克国立大学）；Queens University Belfast（贝尔法斯特女王大学）
Concept drift has potential in smart grid analysis because the socio-economic behaviour of consumers is not governed by the laws of physics. Likewise there are also applications in wind power forecasting. In this paper we present decision tree ensemble classification method based on the Random Forest algorithm for concept drift. The weighted majority voting ensemble aggregation rule is employed based on the ideas of Accuracy Weighted Ensemble (AWE) method. Base learner weight in our case is computed for each sample evaluation using base learners accuracy and intrinsic proximity measure of Rand…
Binary Classification: Is Boosting stronger than Bagging?
Massachusetts Institute of Technology（麻省理工学院）；Operations Research Center, Massachusetts Institute of Technology（麻省理工学院运筹学中心）；Sloan School of Management, Massachusetts Institute of Technology（麻省理工学院斯隆管理学院）
本文提出 Enhanced Random Forests，通过自适应样本权重与个性化树权重，在 15 个二分类数据集上以默认超参数达到或超过 XGBoost，并利用树重要性恢复部分可解释性。
Tencent Youtu Lab（腾讯优图实验室）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Xiamen University（厦门大学）；Peng Cheng Laboratory（鹏城实验室）
This paper proposes a new learning paradigm called filter grafting, which aims to improve the representation capability of Deep Neural Networks (DNNs). The motivation is that DNNs have unimportant (invalid) filters (e.g., l1 norm close to 0). These filters limit the potential of DNNs since they are identified as having little effect on the network. While filter pruning removes these invalid filters for efficiency consideration, filter grafting re-activates them from an accuracy boosting perspective. The activation is processed by grafting external information (weights) into invalid filters. To…
Deep neural networks for image-based screening and computer-aided diagnosis have achieved expert-level performance on various medical imaging modalities, including chest radiographs. Recently, several works have indicated that these state-of-the-art classifiers can be biased with respect to sensitive patient attributes, such as race or gender, leading to growing concerns about demographic disparities and discrimination resulting from algorithmic and model-based decision-making in healthcare. Fair machine learning has focused on mitigating such biases against disadvantaged or marginalised group…
Tencent Youtu Lab（腾讯优图实验室）；Harbin Institute of Technology（哈尔滨工业大学）；XiaMen University（厦门大学）
Filter is the key component in modern convolutional neural networks (CNNs). However, since CNNs are usually over-parameterized, a pre-trained network always contain some invalid (unimportant) filters. These filters have relatively small $l_{1}$ norm and contribute little to the output (\textbf{Reason}). While filter pruning removes these invalid filters for efficiency consideration, we tend to reactivate them to improve the representation capability of CNNs. In this paper, we introduce filter grafting (\textbf{Method}) to achieve this goal. The activation is processed by grafting external info…
Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn features
Massachusetts Institute of Technology（麻省理工学院）；Broad Institute of MIT and Harvard（麻省理工学院和哈佛大学博德研究所）；Halıcıoğlu Data Science Institute, UC San Diego（加州大学圣地亚哥分校哈利奇奥卢数据科学研究所）；Computer Science and Engineering, UC San Diego（加州大学圣地亚哥分校计算机科学与工程系）
Lumberjack: Better Differentially Private Random Forests through Heavy Hitter Detection in Trees
PreMeDICaL；Inria（法国国家信息与自动化研究所）；Université de Montpellier（蒙彼利埃大学）；INSERM（国家医学研究院）；Technical University of Munich（慕尼黑技术大学）
本文提出Lumberjack算法，通过构建大规模随机决策树并应用隐私保护的剪枝技术，显著提升了差分隐私随机森林的实用性。该方法引入了新的（ε，δ）-DP Heavy Hitter检测算法，具有O_{ε,δ}(√log h)的误差，使得树的高度可以更深，从而在隐私约束下提高表达能力。实验表明，Lumberjack在基准数据集上优于现有差分隐私随机森林方法，特别是在隐私预算下的隐私-效用权衡上取得显著改进。
Toward a Robust R2D2 Paradigm for Radio-interferometric Imaging: Revisiting Deep Neural Network Training and Architecture
Institute of Sensors, Signals and Systems, Heriot-Watt University, Edinburgh EH14 4AS, United Kingdom（传感器、信号与系统研究所，赫里奥特-瓦特大学）；EPCC, University of Edinburgh, Potterrow, Edinburgh EH8 9BT, United Kingdom（爱丁堡大学埃平中心）
