arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.16746cs.LGcs.AIcs.CL
收藏

负责任的基础模型开发速查表:工具与资源综述

The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources

MIT(麻省理工学院);EleutherAI;UCSB(加利福尼亚大学圣巴巴拉分校);SynthLabs;Princeton University(普林斯顿大学);Stanford University(斯坦福大学);Harvard University(哈佛大学);Allen Institute for AI(艾伦人工智能研究所);Google DeepMind(谷歌DeepMind);ML Commons;Contextual AI;HuggingFace;University College London(伦敦大学学院)
Shayne Longpre, Stella Biderman, Alon Albalak, Hailey Schoelkopf, Daniel McDuff, Sayash Kapoor, Kevin Klyman, Kyle Lo, Gabriel Ilharco, Nay San, Maribeth Rauh, … 展开作者

Shayne Longpre, Stella Biderman, Alon Albalak, Hailey Schoelkopf, Daniel McDuff, Sayash Kapoor, Kevin Klyman, Kyle Lo, Gabriel Ilharco, Nay San, Maribeth Rauh, Aviya Skowron, Bertie Vidgen, Laura Weidinger, Arvind Narayanan, Victor Sanh, David Adelani, Percy Liang, Rishi Bommasani, Peter Henderson, Sasha Luccioni, Yacine Jernite, Luca Soldaini

AI总结:

本文整理涵盖文本、视觉、语音的 250 余种基础模型开发工具与资源,并据此指出数据、评估、监控、透明度、多语言多模态及系统级评估方面的关键缺口。

摘要:

基础模型开发正吸引着迅速扩大的贡献者、科学家和应用群体。为帮助塑造负责任的开发实践,我们推出 Foundation Model Development Cheatsheet(基础模型开发速查表):这是一个持续扩充的集合,包含 250 多种工具和资源,涵盖文本、视觉和语音模态。我们借鉴大量先前工作,调研了支持以下环节的资源(例如软件、文档、框架、指南和实用工具):知情的数据选择、处理与理解,精确且意识到局限性的制品文档记录,高效的模型训练,提前了解训练带来的环境影响,对能力、风险和声明进行审慎的模型评估,以及负责任的模型发布、许可和部署实践。我们希望这一经过整理的资源集合有助于指导更负责任的开发。整理该清单的过程使我们能够审视 AI 开发生态系统,揭示现有实践中哪些工具严重缺失、被误用或被过度使用。我们发现:(i)用于数据获取、模型评估和监控的工具严重未能满足伦理和现实需求;(ii)模型安全性、能力和环境影响的评估均缺乏可复现性和透明度;(iii)文本分析,尤其是以英语为中心的分析,继续主导多语言和多模态分析;(iv)需要评估系统而不仅仅是模型,以便在具体情境中评估能力和影响。

英文摘要:

Foundation model development attracts a rapidly expanding body of contributors, scientists, and applications. To help shape responsible development practices, we introduce the Foundation Model Development Cheatsheet: a growing collection of 250+ tools and resources spanning text, vision, and speech modalities. We draw on a large body of prior work to survey resources (e.g. software, documentation, frameworks, guides, and practical tools) that support informed data selection, processing, and understanding, precise and limitation-aware artifact documentation, efficient model training, advance awareness of the environmental impact from training, careful model evaluation of capabilities, risks, and claims, as well as responsible model release, licensing and deployment practices. We hope this curated collection of resources helps guide more responsible development. The process of curating this list, enabled us to review the AI development ecosystem, revealing what tools are critically missing, misused, or over-used in existing practices. We find that (i) tools for data sourcing, model evaluation, and monitoring are critically under-serving ethical and real-world needs, (ii) evaluations for model safety, capabilities, and environmental impact all lack reproducibility and transparency, (iii) text and particularly English-centric analyses continue to dominate over multilingual and multi-modal analyses, and (iv) evaluation of systems, rather than just models, is needed so that capabilities and impact are assessed in context.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑