The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources
MIT(麻省理工学院);EleutherAI;UCSB(加利福尼亚大学圣巴巴拉分校);SynthLabs;Princeton University(普林斯顿大学);Stanford University(斯坦福大学);Harvard University(哈佛大学);Allen Institute for AI(艾伦人工智能研究所);Google DeepMind(谷歌DeepMind);ML Commons;Contextual AI;HuggingFace;University College London(伦敦大学学院)
Shayne Longpre, Stella Biderman, Alon Albalak, Hailey Schoelkopf, Daniel McDuff, Sayash Kapoor, Kevin Klyman, Kyle Lo, Gabriel Ilharco, Nay San, Maribeth Rauh, … 展开作者
Shayne Longpre, Stella Biderman, Alon Albalak, Hailey Schoelkopf, Daniel McDuff, Sayash Kapoor, Kevin Klyman, Kyle Lo, Gabriel Ilharco, Nay San, Maribeth Rauh, Aviya Skowron, Bertie Vidgen, Laura Weidinger, Arvind Narayanan, Victor Sanh, David Adelani, Percy Liang, Rishi Bommasani, Peter Henderson, Sasha Luccioni, Yacine Jernite, Luca Soldaini
基础模型开发正吸引着迅速扩大的贡献者、科学家和应用群体。为帮助塑造负责任的开发实践,我们推出 Foundation Model Development Cheatsheet(基础模型开发速查表):这是一个持续扩充的集合,包含 250 多种工具和资源,涵盖文本、视觉和语音模态。我们借鉴大量先前工作,调研了支持以下环节的资源(例如软件、文档、框架、指南和实用工具):知情的数据选择、处理与理解,精确且意识到局限性的制品文档记录,高效的模型训练,提前了解训练带来的环境影响,对能力、风险和声明进行审慎的模型评估,以及负责任的模型发布、许可和部署实践。我们希望这一经过整理的资源集合有助于指导更负责任的开发。整理该清单的过程使我们能够审视 AI 开发生态系统,揭示现有实践中哪些工具严重缺失、被误用或被过度使用。我们发现:(i)用于数据获取、模型评估和监控的工具严重未能满足伦理和现实需求;(ii)模型安全性、能力和环境影响的评估均缺乏可复现性和透明度;(iii)文本分析,尤其是以英语为中心的分析,继续主导多语言和多模态分析;(iv)需要评估系统而不仅仅是模型,以便在具体情境中评估能力和影响。
英文摘要:
Foundation model development attracts a rapidly expanding body of contributors, scientists, and applications. To help shape responsible development practices, we introduce the Foundation Model Development Cheatsheet: a growing collection of 250+ tools and resources spanning text, vision, and speech modalities. We draw on a large body of prior work to survey resources (e.g. software, documentation, frameworks, guides, and practical tools) that support informed data selection, processing, and understanding, precise and limitation-aware artifact documentation, efficient model training, advance awareness of the environmental impact from training, careful model evaluation of capabilities, risks, and claims, as well as responsible model release, licensing and deployment practices. We hope this curated collection of resources helps guide more responsible development. The process of curating this list, enabled us to review the AI development ecosystem, revealing what tools are critically missing, misused, or over-used in existing practices. We find that (i) tools for data sourcing, model evaluation, and monitoring are critically under-serving ethical and real-world needs, (ii) evaluations for model safety, capabilities, and environmental impact all lack reproducibility and transparency, (iii) text and particularly English-centric analyses continue to dominate over multilingual and multi-modal analyses, and (iv) evaluation of systems, rather than just models, is needed so that capabilities and impact are assessed in context.
Open Foundation Models in Healthcare: Challenges, Paradoxes, and Opportunities with GenAI Driven Personalized Prescription
Qatar University(卡塔尔大学);Birmingham City University(伯明翰城市大学);Hamad Bin Khalifa University (HBKU)(哈马德·本·哈利法大学);University of the West of England (UWE)(西英格兰大学)
Engineering Carbon Credits Towards A Responsible FinTech Era: The Practices, Implications, and Future
School of Electrical and Computer Engineering(电气与计算机工程学院);The University of Sydney(悉尼大学);Adelaide Business School(阿德莱德商学院);The University of Adelaide(阿德莱德大学);School of Computer Science and Engineering(计算机科学与工程学院);University of New South Wales Sydney(新南威尔士大学悉尼分校);The University of Sydney Business School(悉尼大学商学院)
Geospatial Foundation Models to Enable Progress on Sustainable Development Goals
Helmholtz-Zentrum Dresden-Rossendorf(德累斯顿-罗斯托克研究所);University of Iceland(冰岛大学);Wuhan University(武汉大学);Wuhan University of Science and Technology(武汉科技大学);Universitat de València(瓦伦西亚大学)
Implementing Trust in Non-Small Cell Lung Cancer Diagnosis with a Conformalized Uncertainty-Aware AI Framework in Whole-Slide Images
The Hong Kong Polytechnic University(香港理工大学);Vanderbilt University Medical Center(范德堡大学医学中心);Hong Kong Baptist University(香港浸会大学);Vanderbilt University(范德堡大学)