估计LLM一致性：用户基准与替代指标
Estimating LLM Consistency: A User Baseline vs Surrogate Metrics
AI总结：
本文提出了一种基于logits的集成方法，用于估计LLM一致性，并展示了其在匹配人类评分方面与现有最佳指标相当。
摘要：
大型语言模型（LLMs）容易产生幻觉且对提示扰动敏感，往往导致生成文本不一致或不可靠。已提出不同方法来缓解此类幻觉和脆弱性，其中之一是测量LLM响应的一致性——模型对响应的信心或在重新采样时生成相似响应的可能性。在先前的工作中，测量LLM响应一致性通常依赖于计算响应在重新采样响应池中的概率，分析内部状态或评估响应的logits。然而，不清楚这些方法在多大程度上能近似用户对LLM响应一致性的感知。为了查明这一点，我们进行了用户研究（n=2976），证明了当前测量LLM响应一致性的方法通常与人类对LLM一致性感知不一致。我们提出了一种基于logits的集成方法来估计LLM一致性，并展示了我们的方法在估计人类对LLM一致性评分方面与表现最好的现有指标相当。我们的结果表明，没有人类评估的方法在估计LLM一致性时足够不完美，值得更广泛地使用带有人类输入的评估；这将避免由于自动化一致性度量的不完美而误判模型的充分性。
英文摘要：
Large language models (LLMs) are prone to hallucinations and sensitive to prompt perturbations, often resulting in inconsistent or unreliable generated text. Different methods have been proposed to mitigate such hallucinations and fragility, one of which is to measure the consistency of LLM responses -- the model's confidence in the response or likelihood of generating a similar response when resampled. In previous work, measuring LLM response consistency often relied on calculating the probability of a response appearing within a pool of resampled responses, analyzing internal states, or evaluating logits of responses. However, it was not clear how well these approaches approximated users' perceptions of consistency of LLM responses. To find out, we performed a user study ($n=2,976$) demonstrating that current methods for measuring LLM response consistency typically do not align well with humans' perceptions of LLM consistency. We propose a logit-based ensemble method for estimating LLM consistency and show that our method matches the performance of the best-performing existing metric in estimating human ratings of LLM consistency. Our results suggest that methods for estimating LLM consistency without human evaluation are sufficiently imperfect to warrant broader use of evaluation with human input; this would avoid misjudging the adequacy of models because of the imperfections of automated consistency metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
