一种分层不精确概率方法用于大型语言模型的可靠性评估
A Hierarchical Imprecise Probability Approach to Reliability Assessment of Large Language Models
AI总结:
本文提出HIP-LLM,一种分层不精确概率框架,用于更准确地评估大型语言模型的可靠性。
摘要:
大型语言模型(LLMs)正被越来越多地部署在多样化领域中,这提高了对严格可靠性评估方法的需求。现有的基于基准的评估主要提供模型在数据集上的准确性描述性统计,提供有限的洞察力来了解LLMs在真实操作条件下的概率行为。本文介绍了HIP-LLM,一种用于建模和推断LLM可靠性层次不精确概率框架。基于软件可靠性工程的基础,HIP-LLM将LLM可靠性定义为在给定的操作配置文件(OP)下,在指定数量的未来任务中无故障运行的概率。HIP-LLM在(子)领域间分层表示依赖关系,使从子领域到系统级的可靠性进行多级推断。HIP-LLM嵌入不精确先验以捕捉认知不确定性,并结合OPs以反映使用情境。它推导出后验可靠性范围,以量化先验和数据之间的不确定性。在多个基准数据集上的实验表明,HIP-LLM比现有基准和最新方法提供了更准确和标准化的可靠性表征。提供了一个公开可访问的HIP-LLM存储库。
英文摘要:
Large Language Models (LLMs) are increasingly deployed across diverse domains, raising the need for rigorous reliability assessment methods. Existing benchmark-based evaluations primarily offer descriptive statistics of model accuracy over datasets, providing limited insight into the probabilistic behavior of LLMs under real operational conditions. This paper introduces HIP-LLM, a Hierarchical Imprecise Probability framework for modeling and inferring LLM reliability. Building upon the foundations of software reliability engineering, HIP-LLM defines LLM reliability as the probability of failure-free operation over a specified number of future tasks under a given Operational Profile (OP). HIP-LLM represents dependencies across (sub-)domains hierarchically, enabling multi-level inference from subdomain to system-level reliability. HIP-LLM embeds imprecise priors to capture epistemic uncertainty and incorporates OPs to reflect usage contexts. It derives posterior reliability envelopes that quantify uncertainty across priors and data. Experiments on multiple benchmark datasets demonstrate that HIP-LLM offers a more accurate and standardized reliability characterization than existing benchmark and state-of-the-art approaches. A publicly accessible repository of HIP-LLM is provided.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。