当模型退役时：开源应用中LLM迁移的实证研究
When the Model Retires: An Empirical Study of LLM Migration in Open-Source Applications
浏览论文内容
中文总结 AI 辅助
本研究通过挖掘GitHub提交，实证分析开源应用中LLM模型退役后的迁移行为，发现82%的迁移发生在关闭日期后，且通告长度显著影响迁移时机，并发布了相关数据集。
中文摘要 AI 辅助
基于商业大语言模型（LLM）API构建的应用程序依赖于模型版本，而提供商按照自己的时间表将其退役，通知期从一年到两周不等。我们探究当模型退役时，应用程序实际会发生什么。我们挖掘GitHub上从OpenAI、Anthropic和Google官方弃用的模型和端点迁移出去的提交，并将每个提交与提供商的公开公告和关闭日期进行匹配。在17,703个非fork仓库的22,555个提交中（2024-2026年），有5,139个匹配到官方事件；两位独立编码员验证了一个分层样本（300个，kappa = 0.89-0.95），我们根据其标签重新加权所有估计。我们发现，估计有82%（95%置信区间79-84）的从退役模型迁移的提交是在关闭日期之后提交的——即在应用程序开始失败之后——无论仓库流行度、先前退役经验或是否存在提供商抽象层，这一比例均如此。该比例与提供商的通告政策相关：Anthropic的60-114天通告对应89%，而OpenAI的Assistants API一年通告对应13%；通告长度每增加e倍，关闭后迁移的几率约降低四分之三。模型标识符在94%的迁移应用中被硬编码，迁移工作量从仅提示应用的中位数6行新增代码到微调应用的近700行不等，且只有8%的迁移更换了提供商。我们发布了数据集和流水线，并讨论了对弃用政策、LLM产品的依赖风险评估以及工具的影响。
英文摘要
Applications built on commercial large language model (LLM) APIs depend on model versions that providers retire on their own schedule, with notice periods ranging from one year to two weeks. We ask what actually happens to applications when a model is retired. We mine GitHub for commits that migrate away from officially deprecated models and endpoints of OpenAI, Anthropic, and Google, matching each commit to the provider's published announcement and shutdown dates. From 22,555 commits in 17,703 non-fork repositories (2024-2026), 5,139 are matched to an official event; two independent coders validated a stratified sample of 300 (kappa = 0.89-0.95), and we reweight all estimates by their labels. We find that an estimated 82% (95% CI 79-84) of migrations away from retired models were committed after the shutdown date - after the application had started failing - regardless of repository popularity, prior retirement experience, or the presence of a provider-abstraction layer. The share tracks the provider's notice policy: 89% for Anthropic's 60-114-day notices versus 13% for OpenAI's one-year Assistants API notice, and each e-fold increase in notice length reduces the odds of post-shutdown migration by about three quarters. Model identifiers are hard-coded in 94% of migrating applications, migration effort scales from a median of 6 added lines for prompt-only applications to nearly 700 for fine-tuned ones, and only 8% of migrations switch provider. We release the dataset and pipeline and discuss implications for deprecation policy, dependency-risk assessment of LLM products, and tooling.
发表机构
- Myongji University（明志大学）
机构由 AI 辅助整理，请以论文原文为准。
