多样性度量：语言模型查询中故障的领域无关代理指标
Diversity Measures: Domain-Independent Proxies for Failure in Language Model Queries
AI总结：
针对大语言模型错误预测依赖领域信息的问题，提出三种基于响应多样性的领域无关错误量化度量，经多数据集多温度实验验证其与故障概率强相关，可应用于少样本提示等场景。
摘要：
大语言模型的错误预测通常依赖领域特定信息。本文提出基于给定提示的响应多样性来量化大语言模型响应错误的度量方法，因此与底层应用无关。我们阐述了如何使用三种此类度量——分别基于熵、Gini不纯度和质心距离。我们在多个数据集和温度设置下开展了一系列实验，证明这些度量与故障概率强相关。此外，我们还给出了实证结果，展示了这些度量如何应用于少样本提示、思维链推理和错误检测。
英文摘要：
Error prediction in large language models often relies on domain-specific information. In this paper, we present measures for quantification of error in the response of a large language model based on the diversity of responses to a given prompt - hence independent of the underlying application. We describe how three such measures - based on entropy, Gini impurity, and centroid distance - can be employed. We perform a suite of experiments on multiple datasets and temperature settings to demonstrate that these measures strongly correlate with the probability of failure. Additionally, we present empirical results demonstrating how these measures can be applied to few-shot prompting, chain-of-thought reasoning, and error detection.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
