使用最长支持子序列进行忠实度评估
Evaluation of Faithfulness Using the Longest Supported Subsequence
AI总结：
本研究提出基于最长支持子序列（LSS）的机器生成文本忠实度评估方法，微调模型生成LSS，新指标与人类评分相关性更强，性能优于现有指标，并公开相关数据集与模型。
摘要：
随着越来越复杂的语言模型不断涌现，其可信度成为了一个关键问题，尤其是在摘要生成和问答这类任务中。由于语言的多样性以及可能的答案种类繁多，确保模型的回复基于上下文且具备忠实度颇具挑战。本文提出了一种评估机器生成文本忠实度的新方法，即计算主张中由上下文支持的最长非连续子串，我们将其称为最长支持子序列（Longest Supported Subsequence，LSS）。我们基于一个全新的人工标注数据集，微调了一个模型以生成LSS。我们提出了一种新的评估方法，并证明与不使用LSS的情况相比，采用LSS时这些指标与人类评分的相关性更强。我们提出的指标在我们的数据集上，相较于当前主流的最先进忠实度指标提升了18%。在一个摘要数据集上，针对六种不同的模型，我们的指标始终优于其他指标。最后，我们使用该指标对比了数种流行的大语言模型（Large Language Models，LLMs）的忠实度。我们公开了为预测LSS构建的人工标注数据集，以及我们用于评估忠实度的微调模型。
英文摘要：
As increasingly sophisticated language models emerge, their trustworthiness becomes a pivotal issue, especially in tasks such as summarization and question-answering. Ensuring their responses are contextually grounded and faithful is challenging due to the linguistic diversity and the myriad of possible answers. In this paper, we introduce a novel approach to evaluate faithfulness of machine-generated text by computing the longest noncontinuous substring of the claim that is supported by the context, which we refer to as the Longest Supported Subsequence (LSS). Using a new human-annotated dataset, we finetune a model to generate LSS. We introduce a new method of evaluation and demonstrate that these metrics correlate better with human ratings when LSS is employed, as opposed to when it is not. Our proposed metric demonstrates an 18% enhancement over the prevailing state-of-the-art metric for faithfulness on our dataset. Our metric consistently outperforms other metrics on a summarization dataset across six different models. Finally, we compare several popular Large Language Models (LLMs) for faithfulness using this metric. We release the human-annotated dataset built for predicting LSS and our fine-tuned model for evaluating faithfulness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
