向量RAG与LLM编写的维基:在小型多领域研究上的预注册比较
Single-Round Vector RAG vs an LLM-Compiled Wiki: A Preregistered Comparison on a Small Multi-Domain Research Corpus
AI总结:
本文通过预注册比较,研究了向量RAG系统与LLM编写的维基在小型多领域研究中的表现,发现两者在跨论文连接、答案组织和成本等方面各有优劣,没有单一系统在所有指标上最优。
摘要:
我们预注册了两种帮助LLM回答问题的方法在小型研究语料库上的比较:单轮向量RAG系统和LLM编写的markdown维基。两种系统使用相同的回答生成模型回答了24篇论文中的13个问题,其答案由盲审的LLM法官评分。维基在连接不同论文的发现方面表现更好,但其在答案组织方面的优势在法官调整后并不显著。RAG通过预注册测试满足了单事实查找问题的要求。干净的查询侧成本结果与预期的维基优势相悖:在测试设置下,维基使用了比RAG更多的查询令牌,因此无法通过更便宜的查询来回收前期构建成本。两个探索性分析改变了我们对结果的解读。首先,按主张层面的引用检查更倾向于维基:其引用的页面更常支持所陈述的精确主张,尽管RAG在整体可信度评分上表现更好。其次,基于分解的RAG变体在较低的LLM令牌成本下恢复了维基在跨论文综合方面的大部分优势,但未能恢复维基在按主张引用支持方面的优势。主要结论是,基于事实的研究综合并非单一能力。系统在组织证据、引用支持每个主张的能力以及运行成本方面可能有所不同。在本研究中,没有架构在所有三个指标上最优。
英文摘要:
We preregistered a comparison of two ways to help an LLM answer questions over a small research corpus: single-round Vector RAG and an LLM-compiled markdown wiki browsed by a tool-using agent. Both answered the same 13 questions over 24 papers with the same answer model, scored by two blinded LLM judges. The three preregistered predictions came out one weakly supported, one supported, and one refuted. The wiki scored much better at connecting findings across papers, but its organization advantage fell below the registered threshold once both judges were combined. RAG met the registered test on single-fact groundedness, though the result was judge-sensitive. The wiki was cheaper to build but spent about 21 times more LLM tokens per query, so no break-even point exists. Exploratory analyses bear on why such comparisons disagree. A decomposition-retrieval variant of RAG reduced most of the wiki's synthesis-score gap at lower token cost. The judges' rank agreement was near zero on holistic groundedness (rho = 0.04) against rho = 0.81 on the most concretely defined criterion. A post-hoc claim-level analysis of citation support was checked against two human annotators on 100 claims. Its scorer agreed with them on 50 to 54% of claims as first run and on 65 to 69% once a truncation error was corrected, short of the rule fixed in advance. On the annotators' labels, the analysis did not establish a citation-support advantage for the wiki. On one model and one small corpus, which system appears to win depends on the retrieval baseline, the scoring method, and the judge, so evaluations should report synthesis, citation support, and cost separately and check automated grounding scorers against human labels.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。