NeurIPS 2020 Competition: Predicting Generalization in Deep Learning
Google Research（谷歌研究院）；University of Toronto（多伦多大学）；Element AI；Microsoft Research（微软研究院）；University Paris-Saclay（巴黎-萨克雷大学）；ChaLearn
Yiding Jiang, Pierre Foret, Scott Yak, Daniel M. Roy, Hossein Mobahi, Gintare Karolina Dziugaite, Samy Bengio, Suriya Gunasekar, Isabelle Guyon, Behnam Neyshabur
备注：
20 pages, 2 figures. Accepted for NeurIPS 2020 Competitions Track. Lead organizer: Yiding Jiang
Understanding generalization in deep learning is arguably one of the most important questions in deep learning. Deep learning has been successfully adopted to a large number of problems ranging from pattern recognition to complex decision making, but many recent researchers have raised many concerns about deep learning, among which the most important is generalization. Despite numerous attempts, conventional statistical learning approaches have yet been able to provide a satisfactory explanation on why deep learning works. A recent line of works aims to address the problem by trying to predict the generalization performance through complexity measures. In this competition, we invite the community to propose complexity measures that can accurately predict generalization of models. A robust and general complexity measure would potentially lead to a better understanding of deep learning's underlying mechanism and behavior of deep models on unseen data, or shed light on better generalization bounds. All these outcomes will be important for making deep learning more robust and reliable.
Indian Institute of Technology, Madras（马德拉斯印度理工学院）
Deep Neural Networks can generalize despite being significantly overparametrized. Recent research has tried to examine this phenomenon from various view points and to provide bounds on the generalization error or measures predictive of the generalization gap based on these viewpoints, such as norm-based, PAC-Bayes based, and margin-based analysis. In this work, we provide an interpretation of generalization from the perspective of quality of internal representations of deep neural networks, based on neuroscientific theories of how the human visual system creates invariant and untangled object …
Instituto Nacional de Astrofísica, Óptica, y Electrónica（墨西哥国家天体物理、光学与电子研究所）；ChaLearn；Paradigm（第四范式）；UPSud/INRIA Université Paris-Saclay（巴黎萨克雷大学巴黎第十一大学/法国国家信息与自动化研究所）；Acadia University（阿卡迪亚大学）；Microsoft Research（微软研究院）；Hong Kong University of Science and Technology（香港科技大学）
We organized a competition on Autonomous Lifelong Machine Learning with Drift that was part of the competition program of NeurIPS 2018. This data driven competition asked participants to develop computer programs capable of solving supervised learning problems where the i.i.d. assumption did not hold. Large data sets were arranged in a lifelong learning and evaluation scenario and CodaLab was used as the challenge platform. The challenge attracted more than 300 participants in its two month duration. This chapter describes the design of the challenge and summarizes its main results.
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Recent papers have shown that sufficiently overparameterized neural networks can perfectly fit even random labels. Thus, it is crucial to understand the underlying reason behind the generalization performance of a network on real-world data. In this work, we propose several measures to predict the generalization error of a network given the training data and its parameters. Using one of these measures, based on noise resilience of the network, we secured 5th position in the predicting generalization in deep learning (PGDL) competition at NeurIPS 2020.
Results of the NeurIPS 2023 Neural MMO Competition on Multi-task Reinforcement Learning
Massachusetts Institute of Technology（麻省理工学院）；CarperAI；Competition Winners（竞赛优胜者）；University of Maryland, College Park（马里兰大学帕克分校）；Plurality Institute（Plurality研究所）
JetBrains Research（JetBrains研究院）；HSE University（高等经济大学）；Saint Petersburg State University（圣彼得堡国立大学）
Black-box optimization is one of the vital tasks in machine learning, since it approximates real-world conditions, in that we do not always know all the properties of a given system, up to knowing almost nothing but the results. This paper describes our approach to solving the black-box optimization challenge at NeurIPS 2020 through learning search space partition for local Bayesian optimization. We describe the task of the challenge as well as our algorithm for low budget optimization that we named \texttt{SPBOpt}. We optimize the hyper-parameters of our algorithm for the competition finals u…
Microsoft Research（微软研究院）；Eedi；Rice University（莱斯大学）
In this competition, participants will address two fundamental causal challenges in machine learning in the context of education using time-series data. The first is to identify the causal relationships between different constructs, where a construct is defined as the smallest element of learning. The second challenge is to predict the impact of learning one construct on the ability to answer questions on other constructs. Addressing these challenges will enable optimisation of students' knowledge acquisition, which can be deployed in a real edtech solution impacting millions of students. Part…
Cogitat Ltd.（科吉塔有限公司）；Imperial College London（帝国理工学院）
Building subject-independent deep learning models for EEG decoding faces the challenge of strong covariate-shift across different datasets, subjects and recording sessions. Our approach to address this difficulty is to explicitly align feature distributions at various layers of the deep learning model, using both simple statistical techniques as well as trainable methods with more representational capacity. This follows in a similar vein as covariance-based alignment methods, often used in a Riemannian manifold context. The methodology proposed herein won first place in the 2021 Benchmarks in …
Are we making progress in unlearning? Findings from the first NeurIPS unlearning competition
Google DeepMind（谷歌DeepMind）；Google Research（谷歌研究院）；University of Warwick（华威大学）；University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；University of Montreal（蒙特利尔大学）；ChaLearn；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Université Paris Saclay（巴黎萨克雷大学）
Driving SMARTS is a regular competition designed to tackle problems caused by the distribution shift in dynamic interaction contexts that are prevalent in real-world autonomous driving (AD). The proposed competition supports methodologically diverse solutions, such as reinforcement learning (RL) and offline learning methods, trained on a combination of naturalistic AD data and open-source simulation platform SMARTS. The two-track structure allows focusing on different aspects of the distribution shift. Track 1 is open to any method and will give ML researchers with different backgrounds an opp…
Université Paris-Saclay（巴黎萨克雷大学）；Tsinghua University（清华大学）；MILA - Québec AI Institute（MILA - 魁北克人工智能研究所）；Polytechnique Montréal（蒙特利尔理工学院）；Universitat de Barcelona（巴塞罗那大学）
We present the design and baseline results for a new challenge in the ChaLearn meta-learning series, accepted at NeurIPS'22, focusing on "cross-domain" meta-learning. Meta-learning aims to leverage experience gained from previous tasks to solve new tasks efficiently (i.e., with better performance, little training data, and/or modest computational resources). While previous challenges in the series focused on within-domain few-shot learning problems, with the aim of learning efficiently N-way k-shot tasks (i.e., N class classification problems with k training examples), this competition challen…
The NeurIPS 2022 Neural MMO Challenge: A Massively Multiagent Competition with Specialization and Trade
Massachusetts Institute of Technology（麻省理工学院）；Stanford University（斯坦福大学）；New York University（纽约大学）；AICrowd；NetEase Games AI Lab（网易游戏人工智能实验室）；Bilibili（哔哩哔哩）；Chengdu Goldwin Electronics Technology Co., Ltd（成都金盘电子科技有限公司）；Advanced Network Technology Laboratory, Beijing University of Posts and Telecommunications（北京邮电大学先进网络技术实验室）；The Hong Kong Chinese University（香港中文大学）；The Hong Kong City University（香港城市大学）
NeurIPS 2024 ML4CFD Competition: Harnessing Machine Learning for Computational Fluid Dynamics in Airfoil Design
IRT SystemX；Ansys Inc（安sys公司）；Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；ISIR；NVIDIA（英伟达）；Institut Jean Le Rond D’Alembert（让·勒朗·达朗贝尔研究所）；Criteo AI Lab（Criteo人工智能实验室）；INRIA Saclay（法国国家信息与自动化研究所萨克雷分部）
Microsoft（微软）；IT University of Copenhagen（哥本哈根信息技术大学）；Pinecone；Meta AI Research（Meta AI研究院）；Carnegie Mellon University（卡内基梅隆大学）；Yandex；Zilliz；University of Maryland（马里兰大学）；Fudan University（复旦大学）；Shanghai Jiao Tong University（上海交通大学）；Baidu（百度）
本文总结了 NeurIPS 2023 Big ANN Challenge 竞赛，该竞赛聚焦过滤搜索、分布外数据及稀疏与流式变体等复杂 ANN 搜索，参赛者在资源受限下取得显著性能提升，推动了该领域发展。
AIcrowd Research, AIcrowd（AIcrowd Research，AIcrowd）；Toyota Research Institute（丰田研究所）；Microsoft Research（微软研究院）；Three Thirds；Institute of Informatics, Universidade Federal do Rio Grande do Sul（南里奥格兰德联邦大学信息学研究所）；Temple University（天普大学）；AIcrowd Community（AIcrowd社区）；DiDi Labs（滴滴实验室）
The NeurIPS 2020 Procgen Competition was designed as a centralized benchmark with clearly defined tasks for measuring Sample Efficiency and Generalization in Reinforcement Learning. Generalization remains one of the most fundamental challenges in deep reinforcement learning, and yet we do not have enough benchmarks to measure the progress of the community on Generalization in Reinforcement Learning. We present the design of a centralized benchmark for Reinforcement Learning which can help measure Sample Efficiency and Generalization in Reinforcement Learning by doing end to end evaluation of t…
Generalization is the ability of a model to predict on unseen domains and is a fundamental task in machine learning. Several generalization bounds, both theoretical and empirical have been proposed but they do not provide tight bounds .In this work, we propose a simple yet effective method to predict the generalization performance of a model by using the concept that models that are robust to augmentations are more generalizable than those which are not. We experiment with several augmentations and composition of augmentations to check the generalization capacity of a model. We also provide a …
This transformation of food delivery businesses to online platforms has gained high attention in recent years. This due to the availability of customizing ordering experiences, easy payment methods, fast delivery, and others. The competition between online food delivery providers has intensified to attain a wider range of customers. Hence, they should have a better understanding of their customers' needs and predict their purchasing decisions. Machine learning has a significant impact on companies' bottom line. They are used to construct models and strategies in industries that rely on big dat…
Although deep neural networks are capable of achieving performance superior to humans on various tasks, they are notorious for requiring large amounts of data and computing resources, restricting their success to domains where such resources are available. Metalearning methods can address this problem by transferring knowledge from related tasks, thus reducing the amount of data and computing resources needed to learn new tasks. We organize the MetaDL competition series, which provide opportunities for research groups all over the world to create and experimentally assess new meta-(deep)learni…
Institute of Advanced Research in Artificial Intelligence (IARAI)（高级人工智能研究院（IARAI））；ETH Zurich（苏黎世联邦理工学院）；Layer 6 AI（Layer 6人工智能公司）；ITMO University（圣彼得堡国立信息技术机械与光学大学）；JetBrains Research（JetBrains研究院）；HSE University（圣彼得堡国立高等经济大学）；Monash University（莫纳什大学）；CSIRO’s Data61（联邦科学与工业研究组织Data61研究所）；Kongju National University（公州国立大学）；Bielefeld University（比勒费尔德大学）；Khalifa University（哈利法大学）；Alchera Data Technologies Ltd（Alchera数据技术有限公司）；HERE Technologies（HERE技术公司）；Silesian University of Technology（西里西亚工业大学）
The IARAI Traffic4cast competitions at NeurIPS 2019 and 2020 showed that neural networks can successfully predict future traffic conditions 1 hour into the future on simply aggregated GPS probe data in time and space bins. We thus reinterpreted the challenge of forecasting traffic conditions as a movie completion task. U-Nets proved to be the winning architecture, demonstrating an ability to extract relevant features in this complex real-world geo-spatial process. Building on the previous competitions, Traffic4cast 2021 now focuses on the question of model robustness and generalizability acros…
University of Helsinki（赫尔辛基大学）；Computer Vision Center, Universitat Autònoma de Barcelona（巴塞罗那自治大学计算机视觉中心）；CISPA Helmholtz Center for Information Security（信息安全赫尔姆霍茨中心）；INRIA（法国国家信息与自动化技术研究所）；Yooz；Carnegie Mellon University（卡内基梅隆大学）；NTT（日本NTT公司）；Institute of Science Tokyo（东京科学研究所）；Department of Computer Science（计算机科学系）；Mphasis AI & Applied Tech Lab at Ashoka, Ashoka University（阿什oka大学人工智能与应用技术实验室）；Koita Centre for Digital Health - Ashoka (KCDH-A)（阿什oka数字健康中心（KCDH-A））；Trivedi School of Biosciences, Ashoka University（阿什oka大学Trivedi生物科学学院）
University of California, San Francisco（加州大学旧金山分校）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
We study generalization in deep learning by appealing to complexity measures originally developed in approximation and information theory. While these concepts are challenged by the high-dimensional and data-defined nature of deep learning, we show that simple vector quantization approaches such as PCA, GMMs, and SVMs capture their spirit when applied layer-wise to deep extracted features giving rise to relatively inexpensive complexity measures that correlate well with generalization performance. We discuss our results in 2020 NeurIPS PGDL challenge.
Microsoft Research（微软研究院）；GSI Technology（GSI科技）；IT University of Copenhagen（哥本哈根信息技术大学）；Meta AI Research（Meta人工智能研究院）；Yandex（杨德克斯）；Carnegie Mellon University（卡内基梅隆大学）；Baidu（百度）
Despite the broad range of algorithms for Approximate Nearest Neighbor Search, most empirical evaluations of algorithms have focused on smaller datasets, typically of 1 million points~\citep{Benchmark}. However, deploying recent advances in embedding based techniques for search, recommendation and ranking at scale require ANNS indices at billion, trillion or larger scale. Barring a few recent papers, there is limited consensus on which algorithms are effective at this scale vis-à-vis their hardware cost. This competition compares ANNS algorithms at billion-scale by hardware cost, accuracy and …
Jožef Stefan Institute（约瑟夫·斯特凡研究所）；Jožef Stefan International Postgraduate School（约瑟夫·斯特凡国际研究生院）
The competition "Predicting Generalization in Deep Learning (PGDL)" aims to provide a platform for rigorous study of generalization of deep learning models and offer insight into the progress of understanding and explaining these models. This report presents the solution that was submitted by the user \emph{smeznar} which achieved the eight place in the competition. In the proposed approach, we create simple metrics and find their best combination with automatic testing on the provided dataset, exploring how combinations of various properties of the input neural network architectures can be us…
The Pommerman Team Environment is a recently proposed benchmark which involves a multi-agent domain with challenges such as partial observability, decentralized execution (without communication), and very sparse and delayed rewards. The inaugural Pommerman Team Competition held at NeurIPS 2018 hosted 25 participants who submitted a team of 2 agents. Our submission nn_team_skynet955_skynet955 won 2nd place of the "learning agents'' category. Our team is composed of 2 neural networks trained with state of the art deep reinforcement learning algorithms and makes use of concepts like reward shapin…
Software verification competitions, such as the annual SV-COMP, evaluate software verification tools with respect to their effectivity and efficiency. Typically, the outcome of a competition is a (possibly category-specific) ranking of the tools. For many applications, such as building portfolio solvers, it would be desirable to have an idea of the (relative) performance of verification tools on a given verification task beforehand, i.e., prior to actually running all tools on the task. In this paper, we present a machine learning approach to predicting rankings of tools on verification tasks.…
University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；University of Maryland, College Park（马里兰大学帕克分校）；AI Labs at Didi Chuxing（滴滴出行人工智能实验室）
Millions of drivers worldwide have enjoyed financial benefits and work schedule flexibility through a ride-sharing economy, but meanwhile they have suffered from the lack of a sense of identity and career achievement. Equipped with social identity and contest theories, financially incentivized team competitions have been an effective instrument to increase drivers' productivity, job satisfaction, and retention, and to improve revenue over cost for ride-sharing platforms. While these competitions are overall effective, the decisive factors behind the treatment effects and how they affect the ou…
In this paper, we study a long-term planning scenario that is based on drone racing competitions held in real life. We conducted this experiment on a framework created for "Game of Drones: Drone Racing Competition" at NeurIPS 2019. The racing environment was created using Microsoft's AirSim Drone Racing Lab. A reinforcement learning agent, a simulated quadrotor in our case, has trained with the Policy Proximal Optimization(PPO) algorithm was able to successfully compete against another simulated quadrotor that was running a classical path planning algorithm. Agent observations consist of data …
The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Google（谷歌）
A large body of work in behavioral fields attempts to develop models that describe the way people, as opposed to rational agents, make decisions. A recent Choice Prediction Competition (2015) challenged researchers to suggest a model that captures 14 classic choice biases and can predict human decisions under risk and ambiguity. The competition focused on simple decision problems, in which human subjects were asked to repeatedly choose between two gamble options. In this paper we present our approach for predicting human decision behavior: we suggest to use machine learning algorithms with fea…
Developing a robust generalization measure for the performance of machine learning models is an important and challenging task. A lot of recent research in the area focuses on the model decision boundary when predicting generalization. In this paper, as part of the "Predicting Generalization in Deep Learning" competition, we analyse the complexity of decision boundaries using the concept of fractal dimension and develop a generalization measure based on that technique.
In this technical report, we present our solution to the Traffic4Cast 2021 Core Challenge, in which participants were asked to develop algorithms for predicting a traffic state 60 minutes ahead, based on the information from the previous hour, in 4 different cities. In contrast to the previously held competitions, this year's challenge focuses on the temporal domain shift in traffic due to the COVID-19 pandemic. Following the past success of U-Net, we utilize it for predicting future traffic maps. Additionally, we explore the usage of pre-trained encoders such as DenseNet and EfficientNet and …
The field of Deep Learning is rich with empirical evidence of human-like performance on a variety of prediction tasks. However, despite these successes, the recent Predicting Generalization in Deep Learning (PGDL) NeurIPS 2020 competition suggests that there is a need for more robust and efficient measures of network generalization. In this work, we propose a new framework for evaluating the generalization capabilities of trained networks. We use perturbation response (PR) curves that capture the accuracy change of a given network as a function of varying levels of training sample perturbation…
Monash University（莫纳什大学）；University of East Anglia（东英吉利大学）；University of California Riverside（加州大学河滨分校）
Time series research has gathered lots of interests in the last decade, especially for Time Series Classification (TSC) and Time Series Forecasting (TSF). Research in TSC has greatly benefited from the University of California Riverside and University of East Anglia (UCR/UEA) Time Series Archives. On the other hand, the advancement in Time Series Forecasting relies on time series forecasting competitions such as the Makridakis competitions, NN3 and NN5 Neural Network competitions, and a few Kaggle competitions. Each year, thousands of papers proposing new algorithms for TSC and TSF have utiliz…
IMT Atlantique（IMT大西洋高等矿业电信学院）；Université Fédérale de Toulouse（图卢兹联邦大学）
Measuring the generalization performance of a Deep Neural Network (DNN) without relying on a validation set is a difficult task. In this work, we propose exploiting Latent Geometry Graphs (LGGs) to represent the latent spaces of trained DNN architectures. Such graphs are obtained by connecting samples that yield similar latent representations at a given layer of the considered DNN. We then obtain a generalization score by looking at how strongly connected are samples of distinct classes in LGGs. This score allowed us to rank 3rd on the NeurIPS 2020 Predicting Generalization in Deep Learning (P…
The 2021 NeurIPS Machine Learning for Combinatorial Optimization (ML4CO) competition was designed with the goal of improving state-of-the-art combinatorial optimization solvers by replacing key heuristic components with machine learning models. The competition's main scientific question was the following: is machine learning a viable option for improving traditional combinatorial optimization solvers on specific problem distributions, when historical data is available? This was motivated by the fact that in many practical scenarios, the data changes only slightly between the repetitions of a c…
Techniques for making future predictions based upon the present and past data, has always been an area with direct application to various real life problems. We are discussing a similar problem in this paper. The problem statement is provided by Kaggle, which also serves as an ongoing competition on the Kaggle platform. In this project, we worked with a challenging time-series dataset consisting of daily sales data, kindly provided by one of the largest Russian software firms - 1C Company. The objective is to predict the total sales for every product and store in the next month given the past …
Baylor College of Medicine（贝勒医学院）；University of Göttingen（哥廷根大学）；University of Tübingen（图宾根大学）
The neural underpinning of the biological visual system is challenging to study experimentally, in particular as the neuronal activity becomes increasingly nonlinear with respect to visual input. Artificial neural networks (ANNs) can serve a variety of goals for improving our understanding of this complex system, not only serving as predictive digital twins of sensory cortex for novel hypothesis generation in silico, but also incorporating bio-inspired architectural motifs to progressively bridge the gap between biological and machine vision. The mouse has recently emerged as a popular model s…
In the nascent days of e-content delivery, having a superior product was enough to give companies an edge against the competition. With today's fiercely competitive market, one needs to be multiple steps ahead, especially when it comes to understanding consumers. Focusing on a large set of web portals owned and managed by a private communications company, we propose methods by which these sites' clickstream data can be used to provide a deep understanding of their visitors, as well as their interests and preferences. We further expand the use of this data to show that it can be effectively use…
Iowa State University（爱荷华州立大学）；Jahangirnagar University（贾汉吉尔纳加尔大学）；Daffodil International University（达福德国际大学）
In recent days, the number of technology enthusiasts is increasing day by day with the prevalence of technological products and easy access to the internet. Similarly, the amount of people working behind this rapid development is rising tremendously. Computer programmers consist of a large portion of those tech-savvy people. Codeforces, an online programming and contest hosting platform used by many competitive programmers worldwide. It is regarded as one of the most standardized platforms for practicing programming problems and participate in programming contests. In this research, we propose…
Microsoft Research（微软研究院）；Princeton University（普林斯顿大学）
Research on generalization bounds for deep networks seeks to give ways to predict test error using just the training dataset and the network parameters. While generalization bounds can give many insights about architecture design, training algorithms, etc., what they do not currently do is yield good predictions for actual test error. A recently introduced Predicting Generalization in Deep Learning competition~\citep{jiang2020neurips} aims to encourage discovery of methods to better predict test error. The current paper investigates a simple idea: can test error be predicted using {\em synthet…
International University of Rabat（拉巴特国际大学）；College of Engineering and Architecture（工程与建筑学院）
This article is an introductory work towards a larger research framework relative to Scientific Prediction. It is a mixed between science and philosophy of science, therefore we can talk about Experimental Philosophy of Science. As a first result, we introduce a new forecasting method based on image completion, named Forecasting Method by Image Inpainting (FM2I). In fact, time series forecasting is transformed into fully images- and signal-based processing procedures. After transforming a time series data into its corresponding image, the problem of data forecasting becomes essentially a probl…
This paper covers predicting high-resolution electricity peak demand features given lower-resolution data. This is a relevant setup as it answers whether limited higher-resolution monitoring helps to estimate future high-resolution peak loads when the high-resolution data is no longer available. That question is particularly interesting for network operators considering replacing high-resolution monitoring predictive models due to economic considerations. We propose models to predict half-hourly minima and maxima of high-resolution (every minute) electricity load data while model inputs are of…
