arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.07799cs.CL
收藏

归因或弃权(不执行):大语言模型作为长文档助手

Attribute or Abstain: Large Language Models as Long Document Assistants

Technical University of Darmstadt(达姆施塔特工业大学);Hessian Center for AI (hessian.AI)(黑森人工智能中心);Ubiquitous Knowledge Processing (UKP) Lab(普适知识处理实验室)

Jan Buchmann, Xiao Liu, Iryna Gurevych

AI总结:

本文提出长文档归因基准LAB,在6类任务和5种LLM上评估不同归因方法,发现一步式引用最适合大型微调模型,额外检索有助于小型提示模型,且未发现归因中的“迷失在中间”现象。

摘要:

大语言模型(LLMs)可以帮助人类处理长文档,但已知会产生幻觉。归因可以增加对LLM响应的信任:LLM提供支持其响应的证据,从而增强可验证性。现有的归因方法仅在RAG(检索增强生成)设置中进行了评估,其中初始检索会混淆LLM的性能。这与长文档设置有本质区别,在长文档设置中检索并非必需,但可能有所帮助。因此,目前缺少针对长文档的归因专项评估。为了填补这一空白,我们提出了LAB,一个包含6个多样化长文档任务并带有归因的基准,并在5种不同规模的LLM上进行了不同归因方法的实验。我们发现,引用(citation),即响应生成和证据提取在一步内完成,对于大型和微调模型表现最佳,而额外的检索对于小型、提示驱动的模型有所帮助。我们研究了归因中是否存在“迷失在中间”(Lost in the Middle)现象,但未发现这一现象。我们还发现,证据质量可以在响应简单的数据集上预测响应质量,但在复杂响应上则不然,因为模型难以为复杂论断提供证据。

英文摘要:

LLMs can help humans working with long documents, but are known to hallucinate. Attribution can increase trust in LLM responses: The LLM provides evidence that supports its response, which enhances verifiability. Existing approaches to attribution have only been evaluated in RAG settings, where the initial retrieval confounds LLM performance. This is crucially different from the long document setting, where retrieval is not needed, but could help. Thus, a long document specific evaluation of attribution is missing. To fill this gap, we present LAB, a benchmark of 6 diverse long document tasks with attribution, and experiments with different approaches to attribution on 5 LLMs of different sizes. We find that citation, i.e. response generation and evidence extraction in one step, performs best for large and fine-tuned models, while additional retrieval can help for small, prompted models. We investigate whether the "Lost in the Middle'' phenomenon exists for attribution, but do not find this. We also find that evidence quality can predict response quality on datasets with simple responses, but not so for complex responses, as models struggle with providing evidence for complex claims.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑