IRB: 自动生成鲁棒事实性基准
IRB: Automated Generation of Robust Factuality Benchmarks
AI总结:
IRB通过自动生成鲁棒事实性基准,评估RAG系统事实性,发现前沿LLM在闭书设置中面临挑战,推理LLM更可靠,改进检索组件更有效。
摘要:
静态的RAG系统基准往往面临快速饱和的问题,并且需要大量手动努力来维持鲁棒性。为了解决这一问题,我们提出了IRB,一个用于自动生成基准以评估RAG系统事实性的框架。IRB采用了一种结构化的生成管道,利用事实支架和算法支架。我们利用IRB构建了一个基准,并评估了前沿的LLM和检索器。我们的结果表明,IRB对前沿LLM在闭书设置中构成了重大挑战。此外,我们的评估表明,推理LLM更为可靠,而改进检索组件可能比扩大生成器更具成本效益,以提高RAG系统正确性。
英文摘要:
Static benchmarks for RAG systems often suffer from rapid saturation and require significant manual effort to maintain robustness. To address this, we present IRB, a framework for automatically generating benchmarks to evaluate the factuality of RAG systems. IRB employs a structured generation pipeline utilizing \textit{factual scaffold} and \textit{algorithmic scaffold}. We utilize IRB to construct a benchmark and evaluate frontier LLMs and retrievers. Our results demonstrate that IRB poses a significant challenge for frontier LLMs in the closed-book setting. Furthermore, our evaluation suggests that reasoning LLMs are more reliable, and that improving the retrieval component may yield more cost-effective gains in RAG system correctness than scaling the generator.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。