The Minimum Information about CLinical Artificial Intelligence Checklist for Generative Modeling Research (MI-CLAIM-GEN)
University of California, San Francisco(加州大学旧金山分校);University of California, Berkeley(加州大学伯克利分校);University of California, Office of the President(加州大学校长办公室);Bayesian Health(贝叶斯健康公司);Johns Hopkins University(约翰斯·霍普金斯大学);Scripps Research(斯克里普斯研究所);University of Chicago(芝加哥大学);Qualified Health PBC(合格健康公益公司)
Brenda Y. Miao, Irene Y. Chen, Christopher YK Williams, Jaysón Davidson, Augusto Garcia-Agundez, Shenghuan Sun, Travis Zack, Suchi Saria, Rima Arnaout, Giorgio … 展开作者
Brenda Y. Miao, Irene Y. Chen, Christopher YK Williams, Jaysón Davidson, Augusto Garcia-Agundez, Shenghuan Sun, Travis Zack, Suchi Saria, Rima Arnaout, Giorgio Quer, Hossein J. Sadaei, Ali Torkamani, Brett Beaulieu-Jones, Bin Yu, Milena Gianfrancesco, Atul J. Butte, Beau Norgeot, Madhumita Sushil
Recent advances in generative models, including large language models (LLMs), vision language models (VLMs), and diffusion models, have accelerated the field of natural language and image processing in medicine and marked a significant paradigm shift in how biomedical models can be developed and deployed. While these models are highly adaptable to new tasks, scaling and evaluating their usage presents new challenges not addressed in previous frameworks. In particular, the ability of these models to produce useful outputs with little to no specialized training data ("zero-" or "few-shot" approaches), as well as the open-ended nature of their outputs, necessitate the development of new guidelines for robust reporting of clinical generative model research. In response to gaps in standards and best practices for the development of clinical AI tools identified by US Executive Order 141103 and several emerging national networks for clinical AI evaluation, we begin to formalize some of these guidelines by building on the original MI-CLAIM checklist. The new checklist, MI-CLAIM-GEN (Table 1), aims to address differences in training, evaluation, interpretability, and reproducibility of new generative models compared to non-generative ("predictive") AI models. This MI-CLAIM-GEN checklist also seeks to clarify cohort selection reporting with unstructured clinical data and adds additional items on alignment with ethical standards for clinical AI research.
Reproducibility in NLP: What Have We Learned from the Checklist?
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院)
Going Beyond Nouns With Vision & Language Models Using Synthetic Data
Rice University(莱斯大学);MIT-IBM Watson AI Lab(麻省理工-IBM沃森人工智能实验室);MIT(麻省理工学院);Georgia Institute of Technology(佐治亚理工学院);Weizmann Institute of Science(魏茨曼科学研究所);IBM Research(IBM研究院);LIGM, École des Ponts(巴黎桥路学院信息学与数学实验室)
Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?
Tohoku University(东洋大学);Research and Development Center for Large Language Models, National Institute of Informatics(大型语言模型研究与开发中心,日本信息处理学会);National Institute of Informatics(日本信息处理学会)
ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists
Computer Science and Engineering, University of Michigan(计算机科学与工程系,密歇根大学);University of Michigan Law School(密歇根大学法学院);School of Information, University of Michigan(信息学院,密歇根大学);Materials Science & Engineering, University of Michigan(材料科学与工程系,密歇根大学);Department of Chemistry, Carnegie Mellon University(化学系,卡内基梅隆大学);Biomedical Engineering, University of Michigan(生物医学工程系,密歇根大学)
Accelerate Creation of Product Claims Using Generative AI
Department of Computer Science University of Cincinnati(计算机科学系 奥克兰大学);Corporate Functions R&D, Discovery& Innovation Platforms, P&G(企业职能研发、发现与创新平台,宝洁公司);Beauty Care IT, P&G(美容护理IT,宝洁公司);Beauty Care R&D, P&G(美容护理研发,宝洁公司)