Data-driven science is an emerging paradigm where scientific discoveries depend on the execution of computational AI models against rich, discipline-specific datasets. With modern machine learning frameworks, anyone can develop and execute computational models that reveal concepts hidden in the data that could enable scientific applications. For important and widely used datasets, computing the performance of every computational model that can run against a dataset is cost prohibitive in terms of cloud resources. Benchmarking approaches used in practice use representative datasets to infer performance without actually executing models. While practicable, these approaches limit extensive dataset profiling to a few datasets and introduce bias that favors models suited for representative datasets. As a result, each dataset's unique characteristics are left unexplored and subpar models are selected based on inference from generalized datasets. This necessitates a new paradigm that introduces dataset profiling into the model selection process. To demonstrate the need for dataset-specific profiling, we answer two questions:(1) Can scientific datasets significantly permute the rank order of computational models compared to widely used representative datasets? (2) If so, could lightweight model execution improve benchmarking accuracy? Taken together, the answers to these questions lay the foundation for a new dataset-aware benchmarking paradigm.
Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization
University of Maryland（马里兰大学）；University of Waterloo（滑铁卢大学）；Carnegie Mellon University（卡内基梅隆大学）；California Institute of Technology（加州理工学院）；Microsoft（微软）；Capital One
Digital Twin Mobility Profiling: A Spatio-Temporal Graph Learning Approach
School of Software, Dalian University of Technology（大连理工大学软件学院）；School of Engineering, IT and Physical Sciences, Federation University Australia（澳大利亚联邦大学工程、信息技术与物理科学学院）
PPA: Preference Profiling Attack Against Federated Learning
Nanjing University of Science and Technology（南京理工大学）；Institute of Information Engineering, Chinese Academy of Science（中国科学院信息工程研究所）；Data61, CSIRO（澳大利亚联邦科学与工业研究组织Data61）；University of Chinese Academy of Science（中国科学院大学）
