Reproducibility Beyond the Research Community: Experience from NLP Beginners
Shane Storks, Keunwoo Peter Yu, Joyce Chai
备注:
This version has been removed by arXiv administrators because the submitter did not have the authority to grant the license for this version at the time of submission
As NLP research attracts public attention and excitement, it becomes increasingly important for it to be accessible to a broad audience. As the research community works to democratize NLP, it remains unclear whether beginners to the field can easily apply the latest developments. To understand their needs, we conducted a study with 93 students in an introductory NLP course, where students reproduced results of recent NLP papers. Surprisingly, our results suggest that their technical skill (i.e., programming experience) has limited impact on their effort spent completing the exercise. Instead, we find accessibility efforts by research authors to be key to a successful experience, including thorough documentation and easy access to required models and datasets.
Reproducibility in NLP: What Have We Learned from the Checklist?
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院)
Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP
ADAPT/DCU(ADAPT中心/都柏林城市大学);University of Aberdeen(阿伯丁大学);Bielefeld University(比勒费尔德大学);Dataminr(Dataminr公司);Edinburgh Napier University(爱丁堡龙比亚大学);Georgia Tech(佐治亚理工学院);Google Research(谷歌研究院);Groningen University(格罗宁根大学);Heriot-Watt University(赫瑞-瓦特大学);Charles University Prague(布拉格查理大学);Tohoku University(东北大学);Peking University(北京大学);Technological University Dublin(都柏林理工大学);Tilburg University(蒂尔堡大学);trivago(trivago公司);University of Applied Sciences Darmstadt(达姆施塔特应用科学大学);University of Illinois Chicago(伊利诺伊大学芝加哥分校);Universidade de Santiago de Compostela(圣地亚哥德孔波斯特拉大学);Universitat Pompeu Fabra(庞培法布拉大学);Utrecht University(乌得勒支大学);Zurich University of Applied Sciences(苏黎世应用科学大学)
Lessons in Reproducibility: Insights from NLP Studies in Materials Science
Cohere(Cohere公司);University of Toronto(多伦多大学);Toyota Research Institute(丰田研究院);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);University of Washington(华盛顿大学)
Systematic Exploration of Dialogue Summarization Approaches for Reproducibility, Comparative Assessment, and Methodological Innovations for Advancing Natural Language Processing in Abstractive Summarization
SRM Institute of Science and Technology(SRM科学技术学院);SRM University, AP(SRM大学(安得拉邦))
Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research
Technical University of Darmstadt(达姆施塔特工业大学);Hessian AI(黑森人工智能中心);Berliner Hochschule für Technik(柏林工程应用技术大学);Osaka University(大阪大学);Brown University(布朗大学);University of Washington(华盛顿大学);IT University of Copenhagen(哥本哈根信息技术大学);Amazon(亚马逊);The Hebrew University of Jerusalem(耶路撒冷希伯来大学);Carnegie Mellon University(卡内基梅隆大学);Allen Institute for AI(艾伦人工智能研究所)
Singapore Management University(新加坡管理大学);CodeX, Stanford University, USA(CodeX,斯坦福大学,美国);Illinois Tech - Chicago Kent College of Law, USA(伊利诺伊理工学院—芝加哥肯特法学院,美国);Bucerius Law School, Germany(伯克勒尔法学院,德国);IIT-Bhubaneswar(IIT-巴纳拉斯瓦尔)
WorkRB: A Community-Driven Evaluation Framework for AI in the Work Domain
University of Copenhagen(哥本哈根大学);Edinburgh Napier University(爱丁堡纳皮尔大学);Leiden University(莱顿大学);University of Amsterdam(阿姆斯特丹大学);Ghent University - imec(根特大学 - imec);AIDA-IDLab, Ghent University(AIDA-IDLab,根特大学);IT University of Copenhagen(哥本哈根IT大学)