发表机构
WMG, University of Warwick; City St George's, University of London; Centre for Software Reliability, City St George's, University of London（华威大学WMG学院; 伦敦城市圣乔治大学; 伦敦城市圣乔治大学软件可靠性中心）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究大语言模型可靠性评估，传统方法有局限。通过放宽独立任务结果假设，引入隐马尔可夫模型扩展分层贝叶斯框架，捕捉顺序依赖性，经实验证明忽略此依赖性会使可靠性估计过度自信。
AI 中文摘要
大语言模型（LLMs）的可靠性评估旨在估计模型在特定操作配置文件下产生正确响应的概率。传统基于基准的评估通常以总体准确率概括，提供性能点估计，但未描述与可靠性声明相关的不确定性。当前，用于LLM可靠性评估的统计推断方法正在兴起。然而，这些模型的一个关键假设是测试结果可视为独立重复试验。在顺序设置中此假设可能不合适，后续响应依赖于通过保留上下文、错误传播或不断演变的交互状态的早期交互。我们通过放宽独立任务结果的假设并引入隐马尔可夫模型来捕捉基准构建交互会话中的顺序依赖性，扩展了用于LLM可靠性评估的分层贝叶斯框架。在此公式中，结果由根据一阶马尔可夫过程演变的潜在交互状态生成，捕捉交互上下文的变化。通过在四个数据集上对Anthropic Claude和OpenAI进行实验，我们证明了顺序依赖性对可靠性评估的潜在影响。结果表明，忽略顺序依赖性可能导致过度自信的可靠性估计。
英文摘要
Reliability assessment of large language models (LLMs) seeks to estimate the probability that a model produces correct responses under a specified operational profile. Conventional benchmark-based evaluation, often summarized by aggregate accuracy, provides a point estimate of performance but does not characterize the uncertainty associated with reliability claims. Currently, statistical inference methods for LLM reliability assessment are emerging. However, a key assumption underlying these models is that test outcomes can be treated as independent repeated trials. This assumption may be inappropriate in sequential settings, where later responses depend on earlier interactions through retained context, error propagation, or an evolving interaction state. We extend a hierarchical Bayesian framework for LLM reliability assessment by relaxing the assumption of independent task outcomes and introducing a Hidden Markov Model to capture sequential dependence in benchmark-constructed interaction sessions. In this formulation, outcomes are generated from a latent interaction state evolving according to a first-order Markov process, capturing changes in interaction context. Through experiments using Anthropic Claude and OpenAI on four datasets, we demonstrate the potential impact of sequential dependence on reliability assessment. The results suggest that ignoring sequential dependence may lead to overconfident reliability estimates.
