arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.19216cs.IR
收藏

大型语言模型擅长效用判断吗?

Are Large Language Models Good at Utility Judgments?

Hengran Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng

AI总结:

本文系统评估了大型语言模型在开放域问答中判断段落效用的能力,揭示了其区分相关性与效用的特性,并提出 k 采样列表式方法以降低对输入顺序的依赖,为检索增强生成提供了基准与改进思路。

摘要:

检索增强生成(RAG)被认为是缓解大型语言模型(LLM)幻觉问题的一种有前景的方法,并且最近受到了研究者的广泛关注。由于检索模型在语义理解上的局限性,RAG 的成功在很大程度上依赖于 LLM 识别具有效用的段落的能力。最近的工作已经探索了 LLM 在检索中评估段落相关性的能力,但在评估段落对支持问答的效用方面的工作仍然有限。在这项工作中,我们对 LLM 在开放域问答中的效用评估能力进行了全面研究。具体来说,我们引入了一个基准测试流程和一组具有不同特征的候选段落集合,从而使用五个具有代表性的 LLM 进行了一系列实验。我们的实验表明:(i)经过良好指令调优的 LLM 能够区分相关性和效用,并且 LLM 对新生成的反事实段落具有很高的接受度。此外,(ii)我们仔细考察了指令设计中影响效用判断的关键因素。最后,(iii)为了验证效用判断在实际检索增强应用中的有效性,我们深入研究了 LLM 在使用经效用判断的证据和直接稠密检索结果时的问答能力。(iv)我们提出了一种 k 采样、列表式的方法,以减少 LLM 对输入段落顺序的依赖,从而促进后续答案生成。我们相信,我们对问题的形式化定义和研究方式以及我们的发现,有助于对检索增强型 LLM 进行批判性评估。我们的代码和基准可以在 \url{https://github.com/ict-bigdatalab/utility_judgments} 找到。

英文摘要:

Retrieval-augmented generation (RAG) is considered to be a promising approach to alleviate the hallucination issue of large language models (LLMs), and it has received widespread attention from researchers recently. Due to the limitation in the semantic understanding of retrieval models, the success of RAG heavily lies on the ability of LLMs to identify passages with utility. Recent efforts have explored the ability of LLMs to assess the relevance of passages in retrieval, but there has been limited work on evaluating the utility of passages in supporting question answering. In this work, we conduct a comprehensive study about the capabilities of LLMs in utility evaluation for open-domain QA. Specifically, we introduce a benchmarking procedure and collection of candidate passages with different characteristics, facilitating a series of experiments with five representative LLMs. Our experiments reveal that: (i) well-instructed LLMs can distinguish between relevance and utility, and that LLMs are highly receptive to newly generated counterfactual passages. Moreover, (ii) we scrutinize key factors that affect utility judgments in the instruction design. And finally, (iii) to verify the efficacy of utility judgments in practical retrieval augmentation applications, we delve into LLMs' QA capabilities using the evidence judged with utility and direct dense retrieval results. (iv) We propose a k-sampling, listwise approach to reduce the dependency of LLMs on the sequence of input passages, thereby facilitating subsequent answer generation. We believe that the way we formalize and study the problem along with our findings contributes to a critical assessment of retrieval-augmented LLMs. Our code and benchmark can be found at \url{https://github.com/ict-bigdatalab/utility_judgments}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑