Time series are generated in diverse domains such as economic, traffic, health, and energy, where forecasting of future values has numerous important applications. Not surprisingly, many forecasting methods are being proposed. To ensure progress, it is essential to be able to study and compare such methods empirically in a comprehensive and reliable manner. To achieve this, we propose TFB, an automated benchmark for Time Series Forecasting (TSF) methods. TFB advances the state-of-the-art by addressing shortcomings related to datasets, comparison methods, and evaluation pipelines: 1) insufficient coverage of data domains, 2) stereotype bias against traditional methods, and 3) inconsistent and inflexible pipelines. To achieve better domain coverage, we include datasets from 10 different domains: traffic, electricity, energy, the environment, nature, economic, stock markets, banking, health, and the web. We also provide a time series characterization to ensure that the selected datasets are comprehensive. To remove biases against some methods, we include a diverse range of methods, including statistical learning, machine learning, and deep learning methods, and we also support a variety of evaluation strategies and metrics to ensure a more comprehensive evaluations of different methods. To support the integration of different methods into the benchmark and enable fair comparisons, TFB features a flexible and scalable pipeline that eliminates biases. Next, we employ TFB to perform a thorough evaluation of 21 Univariate Time Series Forecasting (UTSF) methods on 8,068 univariate time series and 14 Multivariate Time Series Forecasting (MTSF) methods on 25 datasets. The benchmark code and data are available at https://github.com/decisionintelligence/TFB. We have also launched an online time series leaderboard: https://decisionintelligence.github.io/OpenTS/OpenTS-Bench/.
Benchmarking Data Heterogeneity Evaluation Approaches for Personalized Federated Learning
Beijing University of Posts and Telecommunications(北京邮电大学);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);Institute of High Performance Computing, Agency for Science, Technology and Research(新加坡科技研究局高性能计算研究所);Centre for Frontier AI Research, Agency for Science, Technology and Research(新加坡科技研究局前沿人工智能研究中心);School of Engineering, Westlake University(西湖大学工程学院)
Protein-SE(3): Benchmarking SE(3)-based Generative Models for Protein Structure Design
School of Computer Science and Technology, East China Normal University(东华师范大学计算机科学与技术学院);Shanghai Institute of AI for Education, East China Normal University(东华师范大学上海人工智能教育研究所);AI Lab, Research Center for Industries of the Future, Westlake University(西湖大学未来产业研究中心人工智能实验室)
FlexMol: A Flexible Toolkit for Benchmarking Molecular Relational Learning
University of Southern California(南加利福尼亚大学);National University of Singapore(新加坡国立大学);University of Science and Technology of China(中国科学技术大学);Westlake University(西湖大学);Hunan University(湖南大学)
SKADA-Bench: Benchmarking Unsupervised Domain Adaptation Methods with Realistic Validation On Diverse Modalities
École Polytechnique, IP Paris, CMAP, UMR 7641(巴黎理工学院);Université Paris-Saclay, Inria, CEA(巴黎萨克雷大学);Inria(法国国家信息与自动化技术研究院);CEA(法国原子能机构);Centre Borelli, ENS Paris-Saclay(巴黎-萨克雷大学博雷利中心);Università degli Studi di Genova(热那亚大学);Inria, Univ. Rennes 2, CNRS, IRISA(法国国家信息与自动化技术研究院、里昂二大学、法国国家科学研究中心、IRISA)
FairFedMed: Benchmarking Group Fairness in Federated Medical Imaging with FairLoRA
Harvard AI and Robotics Lab(哈佛人工智能与机器人实验室);Harvard University(哈佛大学);Harvard Ophthalmology AI lab(哈佛眼科人工智能实验室);University of Central Florida(佛罗里达中央大学);University of Louisiana at Lafayette(路易斯安那州立大学拉斐特分校);New York University(纽约大学);NYUAD Center for Artificial Intelligence and Robotics(纽约大学阿布扎克人工智能与机器人中心)
Enhancing Classifier Evaluation: A Fairer Benchmarking Strategy Based on Ability and Robustness
Federal University of Pará - UFPA(帕拉联邦大学);Vale Technological Institute - ITV(淡水河谷技术研究所);Federal Institute of Pará - IFPA(帕拉联邦学院);Federal University of Pernambuco - UFPE(伯南布哥联邦大学)
Benchmarking Automated Clinical Language Simplification: Dataset, Algorithm, and Evaluation
College of Information Sciences and Technology, Pennsylvania State University(宾夕法尼亚州立大学信息科学与技术学院);School of Software Technology, Dalian University of Technology(大连理工大学软件学院);Relativity(相对论公司)