大型语言模型公共服务中断与事故的实证特征分析
An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models
AI总结:
本文对 3 家提供商的 8 种公共 LLM 服务进行中断与故障恢复实证分析,揭示恢复时长、周期性和故障隔离等规律,并公开 FAIR 数据与代码。
摘要:
个人和企业越来越依赖 ChatGPT、DALLE 和 Claude 等公共 LLM 服务。了解其中断情况,尤其是衡量其故障恢复过程,正成为一个紧迫问题。然而,这一新兴领域目前仅有有限研究。为解决该问题,本工作对公共 LLM 服务中的中断和故障恢复进行实证特征分析。我们收集并整理了覆盖 3 家主要 LLM 提供商(包括市场领导者 OpenAI 和 Anthropic)的 8 种常用 LLM 服务的数据集。我们详细分析了故障恢复的统计特性、时间模式、共现关系以及引发中断事故的影响范围。我们得出了 10 余项观察结果,其中包括:(1)OpenAI 的 ChatGPT 故障解决时间更长,但发生频率低于 Anthropic 的 Claude;(2)OpenAI 和 Anthropic 的服务故障表现出强烈的周度和月度周期性;(3)OpenAI 服务相比 Anthropic 服务具有更好的故障隔离能力。本研究阐释了 LLM 故障特征,从而支持 LLM 系统构建与使用过程中的优化。FAIR 数据和代码已公开发布于 https://zenodo.org/records/14018219 和 https://github.com/atlarge-research/llm-service-analysis。
英文摘要:
People and businesses increasingly rely on public LLM services, such as ChatGPT, DALLE, and Claude. Understanding their outages, and particularly measuring their failure-recovery processes, is becoming a stringent problem. However, only limited studies exist in this emerging area. Addressing this problem, in this work we conduct an empirical characterization of outages and failure-recovery in public LLM services. We collect and prepare datasets for 8 commonly used LLM services across 3 major LLM providers, including market-leads OpenAI and Anthropic. We conduct a detailed analysis of failure recovery statistical properties, temporal patterns, co-occurrence, and the impact range of outage-causing incidents. We make over 10 observations, among which: (1) Failures in OpenAI's ChatGPT take longer to resolve but occur less frequently than those in Anthropic's Claude;(2) OpenAI and Anthropic service failures exhibit strong weekly and monthly periodicity; and (3) OpenAI services offer better failure-isolation than Anthropic services. Our research explains LLM failure characteristics and thus enables optimization in building and using LLM systems. FAIR data and code are publicly available on https://zenodo.org/records/14018219 and https://github.com/atlarge-research/llm-service-analysis.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。