Algorithm Selection (AS) aims to automatically identify the most suitable optimization algorithm for a given problem instance by leveraging measurable problem characteristics and historical performance data. In this study, we investigate the generalization ability of AS models across both synthetic and real-world optimization landscapes. We consider two widely used academic benchmark suites (BBOB and CEC) and two real-world problem sets (robotics trajectory optimization tasks and unmanned aerial vehicle path-planning problems). Through a systematic cross-benchmark evaluation, we analyze how AS models transfer between domains, identify where generalization succeeds or breaks down, and highlight the challenges that arise when applying AS in realistic, domain-specific contexts. Our findings provide insights into the robustness of current AS approaches and inform the development of more reliable, broadly applicable AS systems for real-world optimization.
The representational capacity of modern neural network architectures has made them a default choice in various applications with high dimensional feature sets. But these high dimensional and potentially noisy features combined with the black box models like neural networks negatively affect the interpretability, generalizability, and the training time of these models. Here, I propose two integrated approaches for feature selection that can be incorporated directly into the parameter learning. One of them involves adding a drop-in layer and performing sequential weight pruning. The other is a s…
The Covid-19 pandemic has caused a dramatic and parallel rise in dangerous misinformation, denoted an `infodemic' by the CDC and WHO. Misinformation tied to the Covid-19 infodemic changes continuously; this can lead to performance degradation of fine-tuned models due to concept drift. Degredation can be mitigated if models generalize well-enough to capture some cyclical aspects of drifted data. In this paper, we explore generalizability of pre-trained and fine-tuned fake news detectors across 9 fake news datasets. We show that existing models often overfit on their training dataset and have po…
University of Calgary（卡尔加里大学）；McGill University Health Centre（麦吉尔大学健康中心）；McGill University（麦吉尔大学）；Massachusetts General Hospital（麻省总医院）；Montreal Imaging Experts（蒙特利尔影像专家公司）；Jewish General Hospital（犹太综合医院）；University of Florida College of Medicine（佛罗里达大学医学院）
Purpose: Despite the potential of machine learning models, the lack of generalizability has hindered their widespread adoption in clinical practice. We investigate three methodological pitfalls: (1) violation of independence assumption, (2) model evaluation with an inappropriate performance indicator or baseline for comparison, and (3) batch effect. Materials and Methods: Using several retrospective datasets, we implement machine learning models with and without the pitfalls to quantitatively illustrate these pitfalls' effect on model generalizability. Results: Violation of independence assump…
Institute of Mathematics and Computer Sciences（数学与计算机科学研究所）；University of São Paulo（圣保罗大学）
With the popularity of Machine Learning (ML) solutions, algorithms and data have been released faster than the capacity of processing them. In this context, the problem of Algorithm Recommendation (AR) is receiving a significant deal of attention recently. This problem has been addressed in the literature as a learning task, often as a Meta-Learning problem where the aim is to recommend the best alternative for a specific dataset. For such, datasets encoded by meta-features are explored by ML algorithms that try to learn the mapping between meta-representations and the best technique to be use…
In this paper, we present and prove some consistency results about the performance of classification models using a subset of features. In addition, we propose to use beam search to perform feature selection, which can be viewed as a generalization of forward selection. We apply beam search to both simulated and real-world data, by evaluating and comparing the performance of different classification models using different sets of features. The results demonstrate that beam search could outperform forward selection, especially when the features are correlated so that they have more discriminati…
University of Minnesota Twin Cities（明尼苏达大学双城分校）；Splunk（Splunk公司）
In the context of supervised parametric models, we introduce the concept of e-values. An e-value is a scalar quantity that represents the proximity of the sampling distribution of parameter estimates in a model trained on a subset of features to that of the model trained on all features (i.e. the full model). Under general conditions, a rank ordering of e-values separates models that contain all essential features from those that do not. The e-values are applicable to a wide range of parametric models. We use data depths and a fast resampling-based algorithm to implement a feature selection pr…
Existing feature selection methods fail to properly account for interactions between features when evaluating feature subsets. In this paper, we attempt to remedy this issue by using orthogonal variance decomposition to evaluate features. The orthogonality of the decomposition allows us to directly calculate the total contribution of a feature to the output variance. Thus we obtain an efficient algorithm for feature evaluation which takes into account interactions among features. Numerical experiments demonstrate that our method accurately identifies relevant features and improves the accuracy…
Evaluating the Generalizability of Foundation Models for Extreme Environmental Events: Case Study of California Wildfire PM2.5
College of Engineering, University of Georgia（佐治亚大学工程学院）；Information System Department, UMBC（马里兰大学巴尔的摩郡分校信息系统系）；College of Graduate and Professional Studies, Trine University（特瑞大学研究生与专业研究学院）；School of Professional Studies, Columbia University（哥伦比亚大学专业研究学院）
Evaluating passing decision-making in professional football: An enhanced MPNN approach to Receiver Selection
Department of Mathematics, Informatics and Geoscience（数学、信息学与地质科学系）；University of Trieste（特里斯特大学）；MathLab（数学实验室）；International School for Advanced Studies (SISSA)（国际高级研究学校（SISSA））；Royal Belgium Football Association（比利时皇家足球协会）
Assessing the Generalizability of a Performance Predictive Model
Jožef Stefan Institute（约瑟夫·斯特凡研究所）；Jožef Stefan International Postgraduate School（约瑟夫·斯特凡国际研究生院）；Leiden University（莱顿大学）；Stellenbosch University（斯泰伦博斯大学）；Gulf University for Science and Technology（海湾科技大学）；Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）
Universidade Federal de Pernambuco（伯南布哥联邦大学）；Instituto Federal da Paraíba（帕拉伊巴联邦学院）；Universidade Federal do Paraná（巴拉那联邦大学）
Dynamic regressor selection (DRS) systems work by selecting the most competent regressors from an ensemble to estimate the target value of a given test pattern. This competence is usually quantified using the performance of the regressors in local regions of the feature space around the test pattern. However, choosing the best measure to calculate the level of competence correctly is not straightforward. The literature of dynamic classifier selection presents a wide variety of competence measures, which cannot be used or adapted for DRS. In this paper, we review eight measures used with regres…
Evaluating Supervised Machine Learning Models: Principles, Pitfalls, and Metric Selection
Jiangsu Key Laboratory of Big Data Security and Intelligent Processing, Nanjing University of Posts and Telecommunications（江苏大数据安全与智能处理重点实验室，南京邮电大学）；University of Brighton（布莱顿大学）；University of Lancashire（兰开夏大学）；School of Cyber Science and Engineering, Southeast University（东南大学信息科学与工程学院）
Budget-Sensitive Discovery Scoring: A Formally Verified Framework for Evaluating AI-Guided Scientific Selection
Indian Institute of Information Technology Allahabad (IIITA)（印度阿瓦德理工学院信息学院（IIITA））；National Institute of Electronics & Information Technology (NIELIT)（国家电子与信息科技研究所（NIELIT））
