In the rapidly evolving domain of Recommender Systems (RecSys), new algorithms frequently claim state-of-the-art performance based on evaluations over a limited set of arbitrarily selected datasets. However, this approach may fail to holistically reflect their effectiveness due to the significant impact of dataset characteristics on algorithm performance. Addressing this deficiency, this paper introduces a novel benchmarking methodology to facilitate a fair and robust comparison of RecSys algorithms, thereby advancing evaluation practices. By utilizing a diverse set of $30$ open datasets, including two introduced in this work, and evaluating $11$ collaborative filtering algorithms across $9$ metrics, we critically examine the influence of dataset characteristics on algorithm performance. We further investigate the feasibility of aggregating outcomes from multiple datasets into a unified ranking. Through rigorous experimental analysis, we validate the reliability of our methodology under the variability of datasets, offering a benchmarking strategy that balances quality and computational demands. This methodology enables a fair yet effective means of evaluating RecSys algorithms, providing valuable guidance for future research endeavors.
ContextGNN goes to Elliot: Towards Benchmarking Relational Deep Learning for Static Link Prediction (aka Personalized Item Recommendation)
University of Barcelona(巴塞罗那大学);University of Southern California(南加州大学);Université Paris-Saclay(巴黎萨克雷大学);CentraleSupélec(中央理工-高等电力学院);Inria(法国国家信息与自动化研究所)
Boosting Revisited: Benchmarking and Advancing LP-Based Ensemble Methods
Industrial Engineering and Management Science University of Twente(特文特大学工业工程与管理科学学院);CIRRELT & SCALE-AI Chair in Data-Driven Supply Chains Polytechnique Montréal(蒙特利尔大学数据驱动供应链联合主席);LAAS-CNRS Université de Toulouse(图卢兹大学CNRS LAAS研究所)
Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments
Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);Huazhong University of Science and Technology(华中科技大学);University of Southern California(南加利福尼亚大学);Midu Technology(米度科技);Northwest University of Political and Law(西北政法大学)
TAME: A Trustworthy Test-Time Evolution of Agent Memory with Systematic Benchmarking
East China Normal University(东华师范大学);Shanghai Innovation Institute(上海创新研究院);Shanghai Key Laboratory of Computer Software Evaluating and Testing(上海计算机软件评测与测试重点实验室);Huawei Noah’s Ark Lab(华为诺亚实验室)
CovDocker: Benchmarking Covalent Drug Design with Tasks, Datasets, and Solutions
Huazhong University of Science and Technology(华中科技大学);Microsoft Research AI for Science(微软研究院人工智能科学部);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking
Rensselaer Polytechnic Institute(罗切斯特理工学院);University of Toronto(多伦多大学);Pennsylvania State University(宾夕法尼亚州立大学);AT&T Chief Data Office(AT&T首席数据办公室);Griffith University(格里菲斯大学)
The Seeker's Dilemma: Realistic Formulation and Benchmarking for Hardware Trojan Detection
Klipsch School of Electrical and Computer Engineering, New Mexico State University(新墨西哥州立大学克利普什电气与计算机工程学院);Department of Computer Systems Technology, North Carolina A&T State University(北卡罗来纳农工州立大学计算机系统技术系);Department of Electrical and Computer Engineering, Miami University(迈阿密大学电气与计算机工程学院)