The emergence of federated learning (FL) presents a promising approach to leverage decentralized data while preserving privacy. Furthermore, the combination of FL and anomaly detection is particularly compelling because it allows for detecting rare and critical anomalies (usually also rare in locally gathered data) in sensitive data from multiple sources, such as cybersecurity and healthcare. However, benchmarking the performance of anomaly detection methods in FL environments remains an underexplored area. This paper introduces FedAD-Bench, a unified benchmark for evaluating unsupervised anomaly detection algorithms within the context of FL. We systematically analyze and compare the performance of recent deep learning anomaly detection models under federated settings, which were typically assessed solely in centralized settings. FedAD-Bench encompasses diverse datasets and metrics to provide a holistic evaluation. Through extensive experiments, we identify key challenges such as model aggregation inefficiencies and metric unreliability. We present insights into FL's regularization effects, revealing scenarios in which it outperforms centralized approaches due to its inherent ability to mitigate overfitting. Our work aims to establish a standardized benchmark to guide future research and development in federated anomaly detection, promoting reproducibility and fair comparison across studies.
Beyond IID: How General Are Tabular Foundation Models, Really?
Prior Labs；University of Freiburg（弗赖堡大学）；University of Mannheim（曼海姆大学）；INRIA Saclay（法国国家信息与自动化研究所萨克雷研究中心）；Technion（技术学院）；ELLIS Institute Tübingen（图宾根ELLIS研究所）；Zuse School ELIZA（Zuse学校ELIZA）
TabArena: A Living Benchmark for Machine Learning on Tabular Data
Amazon Web Services（亚马逊网络服务）；University of Freiburg（弗赖堡大学）；University of Mannheim（曼海姆大学）；INRIA Paris（巴黎国家信息与自动化研究所）；Ecole Normale Supérieure（高等师范学校）；PSL Research University（巴黎科学实验室大学）；Prior Labs（先驱实验室）；ELLIS Institute Tübingen（图宾根ELLIS研究所）
Counterfactual explanations are viewed as an effective way to explain machine learning predictions. This interest is reflected by a relatively young literature with already dozens of algorithms aiming to generate such explanations. These algorithms are focused on finding how features can be modified to change the output classification. However, this rather general objective can be achieved in different ways, which brings about the need for a methodology to test and benchmark these algorithms. The contributions of this work are manifold: First, a large benchmarking study of 10 algorithmic appro…
CLIMB: Class-imbalanced Learning Benchmark on Tabular Data
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；MBZUAI（马克斯·普朗克人工智能研究所）；University of Rochester（罗切斯特大学）；MIT-IBM Watson AI Lab, IBM Research（MIT-IBM沃森人工智能实验室，IBM研究院）；Stony Brook University（石溪大学）；Brookhaven National Laboratory（布鲁赫斯巴赫国家实验室）
VT-Bench: A Unified Benchmark for Visual-Tabular Multi-Modal Learning
National Key Laboratory for Novel Software Technology, Nanjing University, China（新型软件技术国家重点实验室，南京大学，中国）；School of Intelligence Science and Technology, Nanjing University, China（智能科学与技术学院，南京大学，中国）；School of Artificial Intelligence, Nanjing University, China（人工智能学院，南京大学，中国）
University of Freiburg（弗莱堡大学）；Leibniz University Hannover（莱布尼茨汉诺威大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
While early AutoML frameworks focused on optimizing traditional ML pipelines and their hyperparameters, a recent trend in AutoML is to focus on neural architecture search. In this paper, we introduce Auto-PyTorch, which brings the best of these two worlds together by jointly and robustly optimizing the architecture of networks and the training hyperparameters to enable fully automated deep learning (AutoDL). Auto-PyTorch achieves state-of-the-art performance on several tabular benchmarks by combining multi-fidelity optimization with portfolio construction for warmstarting and ensembling of dee…
Unmasking the Chameleons: A Benchmark for Out-of-Distribution Detection in Medical Tabular Data
Amsterdam Public Health Research Institute（阿姆斯特丹公共卫生研究院）；Amsterdam UMC（阿姆斯特丹大学医学中心）；University of Amsterdam（阿姆斯特丹大学）；Institute of Logic, Language and Computation, University of Amsterdam（阿姆斯特丹大学逻辑、语言与计算研究所）；Pacmed（Pacmed公司）
FedAds: A Benchmark for Privacy-Preserving CVR Estimation with Vertical Federated Learning
Alibaba Group（阿里巴巴集团）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
University of Maryland, College Park（马里兰大学帕克分校）；Capital One（第一资本金融公司）
Tabular data underpins numerous high-impact applications of machine learning from fraud detection to genomics and healthcare. Classical approaches to solving tabular problems, such as gradient boosting and random forests, are widely used by practitioners. However, recent deep learning methods have achieved a degree of performance competitive with popular techniques. We devise a hybrid deep learning approach to solving tabular data problems. Our method, SAINT, performs attention over both rows and columns, and it includes an enhanced embedding method. We also study a new contrastive self-superv…
IIIT Hyderabad（印度海得拉巴国际信息技术学院）；Google Research（谷歌研究院）；University of Utah（犹他大学）
Creating challenging tabular inference data is essential for learning complex reasoning. Prior work has mostly relied on two data generation strategies. The first is human annotation, which yields linguistically diverse data but is difficult to scale. The second category for creation is synthetic generation, which is scalable and cost effective but lacks inventiveness. In this research, we present a framework for semi-automatically recasting existing tabular data to make use of the benefits of both approaches. We utilize our framework to build tabular NLI instances from five datasets that were…
Do Quantum Transformers Help? A Systematic VQC Architecture Comparison on Tabular Benchmarks
Beth Israel Deaconess Medical Center \& Harvard Medical School Boston, MA, USA；Department of Electrophysics National Yang Ming Chiao Tung University Hsinchu, Taiwan
Towards Benchmarking Foundation Models for Tabular Data With Text
Department of Computer Science, University of Freiburg, Freiburg, Germany（弗赖堡大学计算机科学系）；Prior Labs, Freiburg, Germany（Prior实验室）；ELLIS Institute, Tubingen, Germany（图宾根ELLIS研究所）
TabAttackBench: A Benchmark for Adversarial Attacks on Tabular Data
School of Information Systems, Queensland University of Technology（昆士兰理工大学信息系统学院）；Center for Data Science, Queensland University of Technology（昆士兰理工大学数据科学中心）；School of Software, Tsinghua University（清华大学软件学院）；NOVA Information Management School, NOVA University of Lisbon（里斯本大学NOVA信息管理学院）；Data Science Institute, University of Technology Sydney（悉尼科技大学数据科学研究院）；INESC-ID/Instituto Superior Técnico, University of Lisboa（里斯本大学INESC-ID/技术高等学院）
Given a labeled training set and a collection of unlabeled data, the goal of active learning (AL) is to identify the best unlabeled points to label. In this comprehensive study, we analyze the performance of a variety of AL algorithms on deep neural networks trained on 69 real-world tabular classification datasets from the OpenML-CC18 benchmark. We consider different data regimes and the effect of self-supervised model pre-training. Surprisingly, we find that the classical margin sampling technique matches or outperforms all others, including current state-of-art, in a wide range of experiment…
SelectArena: A Modern Benchmark for Feature Selection on Real-world Tabular Data
Université du Littoral-Côte-d’Opale（滨海-加来海峡大学）；University of Freiburg（弗莱堡大学）；Prior Labs（Prior实验室）；HTW Berlin（柏林应用科学大学）；ELLIS Institute Tübingen（ELLIS蒂宾根研究所）
University of Freiburg（弗赖堡大学）；University of Mannheim（曼海姆大学）；Bosch Center for AI（博世人工智能中心）
The most significant barrier to the advancement of Neural Architecture Search (NAS) is its demand for large computational resources, which hinders scientifically sound empirical evaluations of NAS methods. Tabular NAS benchmarks have alleviated this problem substantially, making it possible to properly evaluate NAS methods in seconds on commodity machines. However, an unintended consequence of tabular NAS benchmarks has been a focus on extremely small architectural search spaces since their construction relies on exhaustive evaluations of the space. This leads to unrealistic results that do no…
TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments
National Key Laboratory for Novel Software Technology, Nanjing University, China（新型软件技术国家重点实验室，南京大学，中国）；School of Artificial Intelligence, Nanjing University, China（人工智能学院，南京大学，中国）；School of Intelligence Science and Technology, Nanjing University, China（智能科学与技术学院，南京大学，中国）
Yale University（耶鲁大学）；The Chinese University of Hong Kong（香港中文大学）；Northeastern University（东北大学）；Penn State University（宾夕法尼亚州立大学）
Numerical reasoning over hybrid data containing both textual and tabular content (e.g., financial reports) has recently attracted much attention in the NLP community. However, existing question answering (QA) benchmarks over hybrid data only include a single flat table in each document and thus lack examples of multi-step numerical reasoning across multiple hierarchical tables. To facilitate data analytical progress, we construct a new large-scale benchmark, MultiHiertt, with QA pairs over Multi Hierarchical Tabular and Textual data. MultiHiertt is built from a wealth of financial reports and …
Unlock the Potential of Large Language Models for Predictive Tabular Tasks in Data Science with Table-Specific Pretraining
Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；Department of Information Systems and Management Engineering, Southern University of Science and Technology（南方科技大学信息 systems 和管理工程系）
