Despite the recent advances showing that a model pre-trained on large-scale source code data is able to gain appreciable generalization capability, it still requires a sizeable amount of data on the target task for fine-tuning. And the effectiveness of the model generalization is largely affected by the size and quality of the fine-tuning data, which is detrimental for target tasks with limited or unavailable resources. Therefore, cross-task generalization, with the goal of improving the generalization of the model to unseen tasks that have not been seen before, is of strong research and application value.
In this paper, we propose a large-scale benchmark that includes 216 existing code-related tasks. Then, we annotate each task with the corresponding meta information such as task description and instruction, which contains detailed information about the task and a solution guide. This also helps us to easily create a wide variety of ``training/evaluation'' task splits to evaluate the various cross-task generalization capabilities of the model. Then we perform some preliminary experiments to demonstrate that the cross-task generalization of models can be largely improved by in-context learning methods such as few-shot learning and learning from task instructions, which shows the promising prospects of conducting cross-task learning research on our benchmark. We hope that the collection of the datasets and our benchmark will facilitate future work that is not limited to cross-task generalization.
SUB: Benchmarking CBM Generalization via Synthetic Attribute Substitutions
Technical University of Munich, Helmholtz Munich, Munich Center for Machine Learning (MCML)(慕尼黑技术大学、亥姆霍兹慕尼黑、慕尼黑机器学习中心);LTCI, Télécom Paris, Institut Polytechnique de Paris(LTCI、巴黎电信、巴黎理工学院)
OpenECG: Benchmarking ECG Foundation Models with Public 1.2 Million Records
The First Affiliated Hospital, Jiangxi Medical College, Nanchang University(南昌大学江西医学院第一附属医院);Nanchang University(南昌大学);Georgia Institution of Technology(佐治亚理工学院)
Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarking
The Hebrew University of Jerusalem(耶路撒冷希伯来大学);Allen Institute for AI(艾伦人工智能研究院);Bar-Ilan University(巴伊兰大学);Tel-Aviv University(特拉维夫大学);Stanford University(斯坦福大学)
Benchmarking bandgap prediction in semiconductors under experimental and realistic evaluation settings
Centre for Machine Intelligence, University of Sheffield(智能研究中心,谢菲尔德大学);School of Computer Science, University of Sheffield(计算机科学学院,谢菲尔德大学);Climate Office, European Space Agency(气候办公室,欧洲航天局);School of Mathematical and Physical Sciences, University of Sheffield(数学与物理科学学院,谢菲尔德大学);School of Chemical, Materials and Biological Engineering, University of Sheffield(化学、材料与生物工程学院,谢菲尔德大学)
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
University of Waterloo(滑铁卢大学);Georgia Institute of Technology(佐治亚理工学院);Hong Kong University of Science and Technology(香港科技大学);Google DeepMind(谷歌DeepMind)
Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment
Flower Labs;University of Cambridge(剑桥大学);University of Auckland(奥克兰大学);University of Melbourne(墨尔本大学);Vector Institute;Fraunhofer IMS(弗劳恩霍夫应用固体物理与材料力学研究所);NetCompany;Gachon University(嘉泉大学);Owkin;Sony AI(索尼人工智能)