High-accurate machine learning (ML) image classifiers cannot guarantee that they will not fail at operation. Thus, their deployment in safety-critical applications such as autonomous vehicles is still an open issue. The use of fault tolerance mechanisms such as safety monitors is a promising direction to keep the system in a safe state despite errors of the ML classifier. As the prediction from the ML is the core information directly impacting safety, many works are focusing on monitoring the ML model itself. Checking the efficiency of such monitors in the context of safety-critical applications is thus a significant challenge. Therefore, this paper aims at establishing a baseline framework for benchmarking monitors for ML image classifiers. Furthermore, we propose a framework covering the entire pipeline, from data generation to evaluation. Our approach measures monitor performance with a broader set of metrics than usually proposed in the literature. Moreover, we benchmark three different monitor approaches in 79 benchmark datasets containing five categories of out-of-distribution data for image classifiers: class novelty, noise, anomalies, distributional shifts, and adversarial attacks. Our results indicate that these monitors are no more accurate than a random monitor. We also release the code of all experiments for reproducibility.
Beyond Rebalancing: Benchmarking Binary Classifiers Under Class Imbalance Without Rebalancing Techniques
Department of Information Systems and Security, College of Information Technology and Center for Artificial Intelligence and Digital Innovation, United Arab Emirates University(信息系统与安全系,信息技术学院和人工智能与数字创新中心,阿联酋大学);College of Engineering and Technology, American University of the Middle East(工程与技术学院,中东大学)
AI applications in forest monitoring need remote sensing benchmark datasets
University of Cambridge(剑桥大学);University of Oviedo(奥维耶多大学);Swansea University(斯旺西大学);Ghent University(根特大学);Queen Mary University of London(伦敦玛丽女王大学);Norwegian Institute for Bioeconomy Research(挪威生物经济研究所)
A Computer Vision Pipeline for Individual-Level Behavior Analysis: Benchmarking on the Edinburgh Pig Dataset
Department of Animal Science, College of Agriculture and Life Sciences, Cornell University(动物科学系,农业与生命科学学院,康奈尔大学);University of Liège, Gembloux Agro-Bio Tech (ULìege-GxABT)(列日大学,吉尔伯特农业生物技术(ULiège-GxABT))
A multi-schematic classifier-independent oversampling approach for imbalanced datasets
University of Rostock(罗斯托克大学);Leibniz-Institute for Food Systems Biology(莱布尼茨食品系统生物学研究所);Technical University of Munich(慕尼黑工业大学);Royal Holloway, University of London(伦敦大学皇家霍洛威学院)
Generative and reproducible benchmarks for comprehensive evaluation of machine learning classifiers
Institute for Biomedical Informatics, University of Pennsylvania(宾夕法尼亚大学生物医学信息学研究所);AGH University of Science and Technology(AGH科技大学)
了解机器学习(ML)算法的优点和缺点对于确定其应用范围至关重要。在这里,我们介绍了DIverse和GENerative ML Benchmark(DIGEN)-一个综合数据集的集合,用于对二进制结果分类的机器学习算法进行全面,可重复和可解释的基准测试。DIGEN资源由40个数学函数组成,这些函数将连续特征映射到离散端点,以创建合成数据集。这40个函数是使用启发式算法发现的,该算法旨在最大限度地提高多个流行的机器学习算法之间的性能差异,从而为评估和比较新方法提供了有用的测试套件。访问生成函数有助于理解为什么一种方法与其他算法相比表现不佳,从而提供改进的想法。包含大量文档和分析的资源是开源的,可在GitHub上获得。
PreAct-Bench: Benchmarking Predictive Monitoring in LLMs
King’s College London(伦敦国王学院);National University of Singapore(新加坡国立大学);Southern University of Science and Technology(南方科技大学);Thomson Reuters Foundational Research(汤姆森路透基础研究);Imperial College London(伦敦帝国学院);The Alan Turing Institute(艾伦·图灵研究所)
Classifier-Free Diffusion-Based Weakly-Supervised Approach for Health Indicator Derivation in Rotating Machines: Advancing Early Fault Detection and Condition Monitoring
LLM Agents Predict Social Media Reactions but Do Not Outperform Text Classifiers: Benchmarking Simulation Accuracy Using 120K+ Personas of 1511 Humans
Institute for Artificial Intelligence Research and Development of Serbia(塞尔维亚人工智能研究与开发研究所);University of Belgrade, Institute for Philosophy and Social Theory, Digital Society Lab(贝尔格莱德大学,哲学与社会理论研究所,数字社会实验室);Complexity Science Hub, Vienna, Austria(维也纳复杂科学中心);Graz University of Technology, Graz, Austria(技术大学格拉茨,格拉茨,奥地利);University of Novi Sad, Faculty of Philosophy, Novi Sad, Serbia(诺维萨德大学,哲学学院,诺维萨德,塞尔维亚);University of Trier, Trier, Germany(特里尔大学,特里尔,德国);Vrije University Amsterdam, Amsterdam, Netherlands(阿姆斯特丹自由大学,阿姆斯特丹,荷兰)
An Intelligent Decision Support Ensemble Voting Model for Coronary Artery Disease Prediction in Smart Healthcare Monitoring Environments
LASTIMI, High School of Technology Mohammed V University in Rabat Sale, Morocco;LASTIMI, Mohammedia School of Engineers Mohammed V University in Rabat Rabat, Morocco;CISIEV. Faculty of Science;Technology in Marrakech Cadi Ayyad University Marrakech, Morocco