发表机构
Mentomy AI（Mentomy AI）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对RAG仅凭答案正确性评估的不足，提出分层一致性框架（HCF），从语料库、检索上下文和生成答案三个层面审计，识别矛盾证据并给出可解释的一致性得分，实验验证其有效性。
AI 中文摘要
检索增强生成（RAG）通常通过最终答案是否正确来评估。这种测试是不充分的：一个答案可能与参考答案匹配，但产生该答案的上下文包含直接矛盾，使得有争议的证据在仅基于答案的审查和检索相关性评分中不可见。本文提出了分层一致性框架（HCF），这是一种事后、模型无关的审计方法，针对RAG过程的三个不同层面：知识语料库、最终检索到的上下文和生成的答案。HCF将语料库冲突表示为带来源链接的原子事实，从而识别出负责的文档，并返回每个答案一致性得分（ACS）以及支持和矛盾的上下文陈述的解释。我们在涵盖五个领域和100个查询-语料库实例的多个受控语料库上评估HCF。一名人工评估员将每个生成的响应与其提供的真实响应进行比较。结果表明，这三个诊断层面可以分离：具有最高平均检索相似度的语料库具有最低的平均ACS，而结构退化的语料库在语料库层面表现更差，但在答案层面表现更好。最重要的是，HCF在若干情况下识别出矛盾的检索证据，而答案仍然与真实答案匹配。HCF不认证事实真实性；它使支持和质疑答案的证据可检查且可归因。
英文摘要
Retrieval-augmented generation (RAG) is commonly evaluated by whether the final answer is correct. That test is insufficient: an answer can match its reference while the context that produced it contains a direct contradiction, leaving the contested evidence invisible to answer-only review and retrieval relevance scores. This paper presents the Hierarchical Consistency Framework (HCF), a post-hoc, model-agnostic audit of three distinct levels of a RAG process: the knowledge corpus, the final retrieved context, and the generated answer. HCF represents corpus conflicts as source-linked atomic facts, thereby identifying the documents responsible, and returns each Answer Consistency Score (ACS) with an explanation of supporting and contradictory contextual statements. We evaluate HCF on several controlled corpora spanning five domains and 100 query-corpus instances. A human evaluator compares every generated response with its supplied ground-truth response. The results show that the three diagnostic levels can dissociate: the corpus with the highest mean retrieval similarity has the lowest mean ACS, while a structurally degraded corpus performs worse at corpus level but better at answer level. Most importantly, HCF identifies contradictory retrieved evidence in several cases where the answer still matches the ground truth. HCF does not certify factual truth; it makes the evidence supporting and challenging an answer inspectable and attributable.
Comments12 pages, 2 figures, 12 tables, 1 GitHub repository
