Network embedding methods map a network's nodes to vectors in an embedding space, in such a way that these representations are useful for estimating some notion of similarity or proximity between pairs of nodes in the network. The quality of these node representations is then showcased through results of downstream prediction tasks. Commonly used benchmark tasks such as link prediction, however, present complex evaluation pipelines and an abundance of design choices. This, together with a lack of standardized evaluation setups can obscure the real progress in the field. In this paper, we aim to shed light on the state-of-the-art of network embedding methods for link prediction and show, using a consistent evaluation pipeline, that only thin progress has been made over the last years. The newly conducted benchmark that we present here, including 17 embedding methods, also shows that many approaches are outperformed even by simple heuristics. Finally, we argue that standardized evaluation tools can repair this situation and boost future progress in this field.
SUMMARY: Recently, novel machine-learning algorithms have shown potential for predicting undiscovered links in biomedical knowledge networks. However, dedicated benchmarks for measuring algorithmic progress have not yet emerged. With OpenBioLink, we introduce a large-scale, high-quality and highly challenging biomedical link prediction benchmark to transparently and reproducibly evaluate such algorithms. Furthermore, we present preliminary baseline evaluation results. AVAILABILITY AND IMPLEMENTATION: Source code, data and supplementary files are openly available at https://github.com/OpenBioLi…
Minimax linkage was first introduced by Ao et al. [3] in 2004, as an alternative to standard linkage methods used in hierarchical clustering. Minimax linkage relies on distances to a prototype for each cluster; this prototype can be thought of as a representative object in the cluster, hence improving the interpretability of clustering results. Bien and Tibshirani analyzed properties of this method in 2011 [2], popularizing the method within the statistics community. Additionally, they performed comparisons of minimax linkage to standard linkage methods, making use of five data sets and two di…
ContextGNN goes to Elliot: Towards Benchmarking Relational Deep Learning for Static Link Prediction (aka Personalized Item Recommendation)
University of Barcelona（巴塞罗那大学）；University of Southern California（南加州大学）；Université Paris-Saclay（巴黎萨克雷大学）；CentraleSupélec（中央理工-高等电力学院）；Inria（法国国家信息与自动化研究所）
Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；Tsinghua University（清华大学）；School of Computer Science and Engineering, The University of New South Wales（新南威尔士大学计算机科学与工程学院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；Department of Computer Science, Aalborg University（奥尔堡大学计算机科学系）
Microsoft（微软）；University College London（伦敦大学学院）；University of Illinois, UC（伊利诺伊大学厄巴纳-香槟分校）；University of Waterloo（滑铁卢大学）
Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big questions in our field. However, the goal is not simply to identify which run is "best", achieving the top score. The goal is to move the field forward by developing new robust techniques, that work in many different settings, and are adopted in research and practice. This paper uses the MS MARCO and TREC Deep Learning Track as our case study, comparing it to the case of TREC ad hoc ranking in the 1990s. We show how the …
Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；École Normale Supérieure Paris Saclay（巴黎萨克雷高等师范学院）；Extrality（Extrality公司）；Criteo AI Lab（Criteo人工智能实验室）
Recent progress in \emph{Geometric Deep Learning} (GDL) has shown its potential to provide powerful data-driven models. This gives momentum to explore new methods for learning physical systems governed by \emph{Partial Differential Equations} (PDEs) from Graph-Mesh data. However, despite the efforts and recent achievements, several research directions remain unexplored and progress is still far from satisfying the physical requirements of real-world phenomena. One of the major impediments is the absence of benchmarking datasets and common physics evaluation protocols. In this paper, we propose…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）；Vega Institute Foundation（维加研究所基金会）；HSE University（高等经济大学）；Artificial Intelligence Research Institute（人工智能研究院）
RWTH Aachen University（亚琛RWTH大学）；University of Oxford（牛津大学）；Mila – Quebec AI Institute（魁北克AI研究所）；Technion - Israel Institute of Technology（技术学院-以色列理工学院）；ETAS Research University of Stuttgart（斯图加特大学ETAS研究所）；Google Research（谷歌研究）；Microsoft Research（微软研究院）
Extracting relationships and interactions between different biological entities is still an extremely challenging problem but has not received much attention as much as extraction in other generic domains. In addition to the lack of annotated data, low benchmarking is still a major reason for slow progress. In order to fill this gap, we compare multiple existing entity and relation extraction models over a recently introduced public dataset, BioRelEx of sentences annotated with biological entities and relations. Our straightforward benchmarking shows that span-based multi-task architectures li…
Causal relationships are commonly examined in manufacturing processes to support faults investigations, perform interventions, and make strategic decisions. Industry 4.0 has made available an increasing amount of data that enable data-driven Causal Discovery (CD). Considering the growing number of recently proposed CD methods, it is necessary to introduce strict benchmarking procedures on publicly available datasets since they represent the foundation for a fair comparison and validation of different methods. This work introduces two novel public datasets for CD in continuous manufacturing pro…
Node classification and link prediction are widely studied in graph representation learning. While both transductive node classification and link prediction operate over a single input graph, they have so far been studied separately. Node classification models take an input graph with node features and incomplete node labels, and implicitly assume that the graph is relationally complete, i.e., no edges are missing. By contrast, link prediction models are solely motivated by relational incompleteness of the input graphs, and do not typically leverage node features or classes. We propose a unify…
Astex Pharmaceuticals（Astex制药公司）；University of Cambridge（剑桥大学）；The Institute of Science and Technology Austria（奥地利科学技术研究所）
We introduce a machine-learning (ML) framework for high-throughput benchmarking of diverse representations of chemical systems against datasets of materials and molecules. The guiding principle underlying the benchmarking approach is to evaluate raw descriptor performance by limiting model complexity to simple regression schemes while enforcing best ML practices, allowing for unbiased hyperparameter optimization, and assessing learning progress through learning curves along series of synchronized train-test splits. The resulting models are intended as baselines that can inform future method de…
Google Research, Brain Team（谷歌研究院（Brain Team））；University of Toronto（多伦多大学）
Evaluating the general abilities of intelligent agents requires complex simulation environments. Existing benchmarks typically evaluate only one narrow task per environment, requiring researchers to perform expensive training runs on many different environments. We introduce Crafter, an open world survival game with visual inputs that evaluates a wide range of general abilities within a single environment. Agents either learn from the provided reward signal or through intrinsic objectives and are evaluated by semantically meaningful achievements that can be unlocked during each episode, such a…
For many decades, research in speech technologies has focused upon improving reliability. With this now meeting user expectations for a range of diverse applications, speech technology is today omni-present. As result, a focus on security and privacy has now come to the fore. Here, the research effort is in its relative infancy and progress calls for greater, multidisciplinary collaboration with security, privacy, legal and ethical experts among others. Such collaboration is now underway. To help catalyse the efforts, this paper provides a high-level overview of some related research. It targe…
WCEbleedGen: A wireless capsule endoscopy dataset and its benchmarking for automatic bleeding classification, detection, and segmentation
Danube Private University（多瑙河私立大学）；Guru Gobind Singh Indraprastha University（古鲁戈宾德辛德拉普拉斯塔大学）；Indira Gandhi Delhi Technical University for Women（英迪拉甘地德里女子技术大学）；All India Institute of Medical Sciences Delhi（全印度医学科学研究所德里分院）
OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences
University of Science and Technology of China（中国科学技术大学）；Institute of Theoretical Physics, Chinese Academy of Sciences（中国科学院理论物理研究所）；Hefei National Laboratory（合肥国家实验室）；Hefei National Research Center for Physical Sciences at the Microscale（合肥微尺度物质科学国家研究中心）；School of Physical Sciences, University of Science and Technology of China（中国科学技术大学物理学院）；Institute for Advanced Algorithms Research（先进算法研究院）；Endless Frontier（无尽前沿（Endless Frontier））
University of Electronic Science and Technology of China（电子科技大学）；Masakhane
Recent progress in text classification has been focused on high-resource languages such as English and Chinese. For low-resource languages, amongst them most African languages, the lack of well-annotated data and effective preprocessing, is hindering the progress and the transfer of successful methods. In this paper, we introduce two news datasets (KINNEWS and KIRNEWS) for multi-class classification of news articles in Kinyarwanda and Kirundi, two low-resource African languages. The two languages are mutually intelligible, but while Kinyarwanda has been studied in Natural Language Processing (…
QuantBench: Benchmarking AI Methods for Quantitative Investment
The Hong Kong University of Science and Technology（香港科学与技术大学）；IDEA Research（IDEA研究院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Beijing Institute of Technology（北京理工大学）
University of Freiburg（弗赖堡大学）；Amazon Cambridge（亚马逊剑桥）；University of Cambridge（剑桥大学）
Despite the recent progress in hyperparameter optimization (HPO), available benchmarks that resemble real-world scenarios consist of a few and very large problem instances that are expensive to solve. This blocks researchers and practitioners not only from systematically running large-scale comparisons that are needed to draw statistically significant results but also from reproducing experiments that were conducted before. This work proposes a method to alleviate these issues by means of a meta-surrogate model for HPO tasks trained on off-line generated data. The model combines a probabilisti…
Entity alignment is to find identical entities in different knowledge graphs (KGs) that refer to the same real-world object. Embedding-based entity alignment techniques have been drawing a lot of attention recently because they can help solve the issue of symbolic heterogeneity in different KGs. However, in this paper, we show that the progress made in the past was due to biased and unchallenging evaluation. We highlight two major flaws in existing datasets that favor embedding-based entity alignment techniques, i.e., the isomorphic graph structures in relation triples and the weak heterogenei…
The University of Adelaide（阿德莱德大学）；Nanjing University（南京大学）；Leiden University（莱顿大学）
Submodular functions play a key role in the area of optimization as they allow to model many real-world problems that face diminishing returns. Evolutionary algorithms have been shown to obtain strong theoretical performance guarantees for a wide class of submodular problems under various types of constraints while clearly outperforming standard greedy approximation algorithms. This paper introduces a setup for benchmarking algorithms for submodular optimization problems with the aim to provide researchers with a framework to enhance and compare the performance of new algorithms for submodular…
The ability to detect objects regardless of image distortions or weather conditions is crucial for real-world applications of deep learning like autonomous driving. We here provide an easy-to-use benchmark to assess how object detection models perform when image quality degrades. The three resulting benchmark datasets, termed Pascal-C, Coco-C and Cityscapes-C, contain a large variety of image corruptions. We show that a range of standard object detection models suffer a severe performance loss on corrupted images (down to 30--60\% of the original performance). However, a simple data augmentati…
Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）
Time-critical analysis of social media streams is important for humanitarian organizations for planing rapid response during disasters. The \textit{crisis informatics} research community has developed several techniques and systems for processing and classifying big crisis-related data posted on social media. However, due to the dispersed nature of the datasets used in the literature (e.g., for training models), it is not possible to compare the results and measure the progress made towards building better models for crisis informatics tasks. In this work, we attempt to bridge this gap by comb…
4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes
Johns Hopkins University（约翰斯·霍普金斯大学）；Stanford University（斯坦福大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Massachusetts Institute of Technology（麻省理工学院）
Harvard University（哈佛大学）；Samsung Semiconductor, Inc.（三星半导体公司）；Syntiant（Syntiant公司）；University of North Carolina, Charlotte（北卡罗来纳大学夏洛特分校）；Cisco Systems（思科系统公司）；California State Polytechnic University, Pomona（加州州立理工大学波莫纳分校）；Real World Insights（Real World Insights公司）；dividiti（dividiti公司）；University of California, Berkeley（加州大学伯克利分校）；Google（谷歌）；STMicroelectronics, Italy（意法半导体（意大利））；Arizona State University（亚利桑那州立大学）；Reality AI（Reality AI公司）；Arm ML Research Lab（Arm机器学习研究实验室）；KU Leuven（鲁汶大学）；Interuniversity Microelectronics Centre (IMEC)（校际微电子中心（IMEC））；University of York（约克大学）
Recent advancements in ultra-low-power machine learning (TinyML) hardware promises to unlock an entirely new class of smart applications. However, continued progress is limited by the lack of a widely accepted benchmark for these systems. Benchmarking allows us to measure and thereby systematically compare, evaluate, and improve the performance of systems and is therefore fundamental to a field reaching maturity. In this position paper, we present the current landscape of TinyML and discuss the challenges and direction towards developing a fair and useful hardware benchmark for TinyML workload…
SeqBench: Benchmarking Sequential Narrative Generation in Text-to-Video Models
University of Michigan（密歇根大学）；Northeastern University（东北大学）；University of Washington（华盛顿大学）；Harvard University（哈佛大学）；Beijing Jiaotong University（北京交通大学）；University of Rochester（罗切斯特大学）；School of Earth Sciences（地球科学学院）
Tsinghua University（清华大学）；Chinese Academy of Sciences（中国科学院）；Alibaba Group（阿里巴巴集团）；University of Pennsylvania（宾夕法尼亚大学）；Microsoft（微软）
Heterogeneous graph neural networks (HGNNs) have been blossoming in recent years, but the unique data processing and evaluation setups used by each work obstruct a full understanding of their advancements. In this work, we present a systematical reproduction of 12 recent HGNNs by using their official codes, datasets, settings, and hyperparameters, revealing surprising findings about the progress of HGNNs. We find that the simple homogeneous GNNs, e.g., GCN and GAT, are largely underestimated due to improper settings. GAT with proper inputs can generally match or outperform all existing HGNNs a…
