一种受哲学相关性启发的迭代效用判断框架
An Iterative Utility Judgment Framework Inspired by Philosophical Relevance via LLMs
AI总结：
本文提出ITEM框架，通过迭代提升信息检索中相关性排序、效用判断和答案生成的性能，实验表明在多个数据集上均优于基线模型。
摘要：
相关性与效用是评估信息检索系统有效性的两个常用指标。相关性强调检索结果与查询的相关性，而效用指结果对信息寻求者的价值。在检索增强生成（RAG）中，高效用结果应优先提供给LLM，因为其输入带宽有限。重新审视RAG的三个核心组件——相关性排序、效用判断和答案生成——与Schutz的哲学相关性系统一致，该系统包含三种相关性类型，代表不同层次的人类认知并相互增强。这些RAG组件也反映了LLM在问答中的三个认知层次。因此，我们提出迭代效用判断框架（ITEM）以促进RAG的各个步骤。我们在检索（TREC DL，WebAP）、效用判断任务（GTI-NQ）和事实性问答（NQ）数据集上进行了广泛实验。实验结果表明，ITEM在效用判断、排序和答案生成方面均优于代表性基线模型。
英文摘要：
Relevance and utility are two frequently used measures to evaluate the effectiveness of an information retrieval (IR) system. Relevance emphasizes the aboutness of a result to a query, while utility refers to the result's usefulness or value to an information seeker. In retrieval-augmented generation (RAG), high-utility results should be prioritized to feed to LLMs due to their limited input bandwidth. Re-examining RAG's three core components-relevance ranking derived from retrieval models, utility judgments, and answer generation-aligns with Schutz's philosophical system of relevances, which encompasses three types of relevance representing different levels of human cognition that enhance each other. These three RAG components also reflect three cognitive levels for LLMs in question-answering. Therefore, we propose an Iterative utiliTy judgmEnt fraMework (ITEM) to promote each step in RAG. We conducted extensive experiments on retrieval (TREC DL, WebAP), utility judgment task (GTI-NQ), and factoid question-answering (NQ) datasets. Experimental results demonstrate improvements of ITEM in utility judgments, ranking, and answer generation upon representative baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
