Analyzing the Machine Learning Conference Review Process
University of California, Berkeley(加州大学伯克利分校);Princeton University(普林斯顿大学);Centennial High School(百年高中);Harvard University(哈佛大学);University of Maryland, College Park(马里兰大学帕克分校)
David Tran, Alex Valtchanov, Keshav Ganapathy, Raymond Feng, Eric Slud, Micah Goldblum, Tom Goldstein
备注:
NeurIPS Workshop on Navigating the Broader Impacts of AI Research. Full version at arXiv:2010.05137
Mainstream machine learning conferences have seen a dramatic increase in the number of participants, along with a growing range of perspectives, in recent years. Members of the machine learning community are likely to overhear allegations ranging from randomness of acceptance decisions to institutional bias. In this work, we critically analyze the review process through a comprehensive study of papers submitted to ICLR between 2017 and 2020. We quantify reproducibility/randomness in review scores and acceptance decisions, and examine whether scores correlate with paper impact. Our findings suggest strong institutional bias in accept/reject decisions, even after controlling for paper quality. Furthermore, we find evidence for a gender gap, with female authors receiving lower scores, lower acceptance rates, and fewer citations per paper than their male counterparts. We conclude our work with recommendations for future conference organizers.
An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process
University of California, Berkeley(加州大学伯克利分校);Princeton University(普林斯顿大学);Centennial High School(百年高中);Harvard University(哈佛大学);University of Maryland, College Park(马里兰大学帕克分校);University of Maryland College Park(马里兰大学帕克分校)
Design and Analysis of the NIPS 2016 Review Process
Carnegie Mellon University(卡内基梅隆大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Max Planck Institute for Software Systems(马克斯·普朗克软件系统研究所);Universite Paris-Saclay(巴黎萨克雷大学);ChaLearn(ChaLearn(查尔恩))
Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews
Department of Computer Science, Stanford University(斯坦福大学计算机科学系);Machine Learning Department, NEC Labs America(NEC美国实验室机器学习部门);Department of Biomedical Data Science, Stanford University(斯坦福大学生物医学数据科学系);Department of Electrical Engineering, Stanford University(斯坦福大学电气工程系);Graduate School of Education, Stanford University(斯坦福大学教育研究生院);Department of Sociology, Stanford University(斯坦福大学社会学系);Graduate School of Business, Stanford University(斯坦福大学商学院);Department of Management Science and Engineering, Stanford University(斯坦福大学管理科学与工程系);Department of Computer Science, UC Santa Barbara(加州大学圣芭芭拉分校计算机科学系)
The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review
University of Pennsylvania(宾夕法尼亚大学);Massachusetts Institute of Technology(麻省理工学院);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);New York University(纽约大学);Princeton University(普林斯顿大学)