Are we making progress in unlearning? Findings from the first NeurIPS unlearning competition
Google DeepMind（谷歌DeepMind）；Google Research（谷歌研究院）；University of Warwick（华威大学）；University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）；University of Montreal（蒙特利尔大学）；ChaLearn；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Université Paris Saclay（巴黎萨克雷大学）
Eleni Triantafillou, Peter Kairouz, Fabian Pedregosa, Jamie Hayes, Meghdad Kurmanji, Kairan Zhao, Vincent Dumoulin, Julio Jacques Junior, Ioannis Mitliagkas, Ju… 展开作者
Eleni Triantafillou, Peter Kairouz, Fabian Pedregosa, Jamie Hayes, Meghdad Kurmanji, Kairan Zhao, Vincent Dumoulin, Julio Jacques Junior, Ioannis Mitliagkas, Jun Wan, Lisheng Sun Hosoya, Sergio Escalera, Gintare Karolina Dziugaite, Peter Triantafillou, Isabelle Guyon
We present the findings of the first NeurIPS competition on unlearning, which sought to stimulate the development of novel algorithms and initiate discussions on formal and robust evaluation methodologies. The competition was highly successful: nearly 1,200 teams from across the world participated, and a wealth of novel, imaginative solutions with different characteristics were contributed. In this paper, we analyze top solutions and delve into discussions on benchmarking unlearning, which itself is a research problem. The evaluation methodology we developed for the competition measures forgetting quality according to a formal notion of unlearning, while incorporating model utility for a holistic evaluation. We analyze the effectiveness of different instantiations of this evaluation framework vis-a-vis the associated compute cost, and discuss implications for standardizing evaluation. We find that the ranking of leading methods remains stable under several variations of this framework, pointing to avenues for reducing the cost of evaluation. Overall, our findings indicate progress in unlearning, with top-performing competition entries surpassing existing algorithms under our evaluation framework. We analyze trade-offs made by different algorithms and strengths or weaknesses in terms of generalizability to new datasets, paving the way for advancing both benchmarking and algorithm development in this important area.
Results of the NeurIPS 2023 Neural MMO Competition on Multi-task Reinforcement Learning
Massachusetts Institute of Technology（麻省理工学院）；CarperAI；Competition Winners（竞赛优胜者）；University of Maryland, College Park（马里兰大学帕克分校）；Plurality Institute（Plurality研究所）
AutoML @ NeurIPS 2018 challenge: Design and Results
Instituto Nacional de Astrofísica, Óptica, y Electrónica（墨西哥国家天体物理、光学与电子研究所）；ChaLearn；Paradigm（第四范式）；UPSud/INRIA Université Paris-Saclay（巴黎萨克雷大学巴黎第十一大学/法国国家信息与自动化研究所）；Acadia University（阿卡迪亚大学）；Microsoft Research（微软研究院）；Hong Kong University of Science and Technology（香港科技大学）
The NeurIPS 2022 Neural MMO Challenge: A Massively Multiagent Competition with Specialization and Trade
Massachusetts Institute of Technology（麻省理工学院）；Stanford University（斯坦福大学）；New York University（纽约大学）；AICrowd；NetEase Games AI Lab（网易游戏人工智能实验室）；Bilibili（哔哩哔哩）；Chengdu Goldwin Electronics Technology Co., Ltd（成都金盘电子科技有限公司）；Advanced Network Technology Laboratory, Beijing University of Posts and Telecommunications（北京邮电大学先进网络技术实验室）；The Hong Kong Chinese University（香港中文大学）；The Hong Kong City University（香港城市大学）
Traffic4cast at NeurIPS 2021 -- Temporal and Spatial Few-Shot Transfer Learning in Gridded Geo-Spatial Processes
Institute of Advanced Research in Artificial Intelligence (IARAI)（高级人工智能研究院（IARAI））；ETH Zurich（苏黎世联邦理工学院）；Layer 6 AI（Layer 6人工智能公司）；ITMO University（圣彼得堡国立信息技术机械与光学大学）；JetBrains Research（JetBrains研究院）；HSE University（圣彼得堡国立高等经济大学）；Monash University（莫纳什大学）；CSIRO’s Data61（联邦科学与工业研究组织Data61研究所）；Kongju National University（公州国立大学）；Bielefeld University（比勒费尔德大学）；Khalifa University（哈利法大学）；Alchera Data Technologies Ltd（Alchera数据技术有限公司）；HERE Technologies（HERE技术公司）；Silesian University of Technology（西里西亚工业大学）
NeurIPS 2024 ML4CFD Competition: Harnessing Machine Learning for Computational Fluid Dynamics in Airfoil Design
IRT SystemX；Ansys Inc（安sys公司）；Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；ISIR；NVIDIA（英伟达）；Institut Jean Le Rond D’Alembert（让·勒朗·达朗贝尔研究所）；Criteo AI Lab（Criteo人工智能实验室）；INRIA Saclay（法国国家信息与自动化研究所萨克雷分部）
Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmark
AIcrowd Research, AIcrowd（AIcrowd Research，AIcrowd）；Toyota Research Institute（丰田研究所）；Microsoft Research（微软研究院）；Three Thirds；Institute of Informatics, Universidade Federal do Rio Grande do Sul（南里奥格兰德联邦大学信息学研究所）；Temple University（天普大学）；AIcrowd Community（AIcrowd社区）；DiDi Labs（滴滴实验室）
Microsoft（微软）；IT University of Copenhagen（哥本哈根信息技术大学）；Pinecone；Meta AI Research（Meta AI研究院）；Carnegie Mellon University（卡内基梅隆大学）；Yandex；Zilliz；University of Maryland（马里兰大学）；Fudan University（复旦大学）；Shanghai Jiao Tong University（上海交通大学）；Baidu（百度）
本文总结了 NeurIPS 2023 Big ANN Challenge 竞赛，该竞赛聚焦过滤搜索、分布外数据及稀疏与流式变体等复杂 ANN 搜索，参赛者在资源受限下取得显著性能提升，推动了该领域发展。
Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification
University of Helsinki（赫尔辛基大学）；Computer Vision Center, Universitat Autònoma de Barcelona（巴塞罗那自治大学计算机视觉中心）；CISPA Helmholtz Center for Information Security（信息安全赫尔姆霍茨中心）；INRIA（法国国家信息与自动化技术研究所）；Yooz；Carnegie Mellon University（卡内基梅隆大学）；NTT（日本NTT公司）；Institute of Science Tokyo（东京科学研究所）；Department of Computer Science（计算机科学系）；Mphasis AI & Applied Tech Lab at Ashoka, Ashoka University（阿什oka大学人工智能与应用技术实验室）；Koita Centre for Digital Health - Ashoka (KCDH-A)（阿什oka数字健康中心（KCDH-A））；Trivedi School of Biosciences, Ashoka University（阿什oka大学Trivedi生物科学学院）
Results of the NeurIPS'21 Challenge on Billion-Scale Approximate Nearest Neighbor Search
Microsoft Research（微软研究院）；GSI Technology（GSI科技）；IT University of Copenhagen（哥本哈根信息技术大学）；Meta AI Research（Meta人工智能研究院）；Yandex（杨德克斯）；Carnegie Mellon University（卡内基梅隆大学）；Baidu（百度）
University of Freiburg（弗莱堡大学）；Leibniz University Hannover（汉诺威莱布尼茨大学）；Leiden University（莱顿大学）；Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Learning to run a Power Network Challenge: a Retrospective Analysis
Reseau de Transport d’Electricite (RTE)（法国电网公司（RTE））；Google Research（谷歌研究院）；Electric Power Research Institute (EPRI)（电力研究院（EPRI））；University College of London（伦敦大学学院）；TenneT（TenneT电网公司）；American University of Beirut（贝鲁特美国大学）；University Paris-Saclay（巴黎萨克雷大学）；CELEC EP Transelectric（CELEC EP Transelectric公司）
