模型退役会给生物医学AI出版物带来可复现性风险

中文摘要 AI 辅助

背景：大型语言模型（LLMs）正以快速且不断加速的速度应用于生物医学研究，然而，承载许多广泛使用模型的商业服务会按弃用计划运行，这可能使科学可复现性变得复杂。方法：我们在PubMed中检索了2022年至2026年3月期间将特定大型语言模型应用于生物医学任务的原创研究文章。一个提取智能体从61077篇文章摘要中识别出模型名称，并有人类审稿人验证了部分内容的提取准确性。提取的模型名称被标准化为规范模型标识符。我们为使用频率最高的50个模型编制了生命周期数据（发布日期、退役日期、状态）。结果：在将分析限制于使用频率最高的50个模型后，我们确定了跨越5242篇独特出版物的8931篇论文-模型提及。在这些提及中，77.7%引用了商业闭源权重模型。总体而言，42%的提及涉及的模型在正式发表时已退役，或计划在发表后两年内退役。从发表到模型退役的中位间隔为538天。结论：许多使用大型语言模型的生物医学出版物在发表后正走向计算不可复现性。模型弃用应被视为生物医学研究的核心报告和保存问题。

英文摘要

Background. Large language models (LLMs) are being adopted in biomedical research at a rapid and accelerating pace, yet commercial services that host many widely used models operate under deprecation schedules that can complicate scientific reproducibility. Methods. We searched PubMed for original research articles from 2022 through March 2026 that applied a specific LLM to a biomedical task. An extraction agent identified model names from 61,077 article abstracts with human reviewers validating a subset for extraction accuracy. Extracted model names were normalized to canonical model identifiers. Lifecycle data (release date, retirement date, status) were compiled for the 50 most frequently used models. Results. We identified 8,931 paper-model mentions spanning 5,242 unique publications after restricting the analysis to the 50 most frequently used models. Among these mentions, 77.7% cited a commercial closed-weight model. Overall, 42% involved a model that was already retired by the time of official publication or is scheduled to retire within two years of publication. The median interval from publication to model retirement was 538 days. Conclusion. Many biomedical publications using LLMs are on a trajectory toward computational non-reproducibility after publication. Model deprecation should be treated as a core reporting and preservation issue for biomedical research.

发表机构

• Medical College of Wisconsin（威斯康星医学院）
• Marquette University（马凯特大学）

机构由 AI 辅助整理，请以论文原文为准。