QO-Bench: 诊断类型化事件元组上的查询操作符保持检索
QO-Bench: Diagnosing Query-Operator-Preserving Retrieval over Typed Event Tuples
AI总结：
提出QO-Bench基准，通过类型化事件元组上的确定性评估，诊断检索增强生成系统在查询操作符（如连接、交集）上的执行瓶颈。
摘要：
许多关于商业、法律和科学语料库的现实世界问题是文本中潜在记录的数据库风格查询的自然语言版本。现有的检索增强生成（RAG）系统主要针对语义相关性进行优化，但检索到看似相关的段落并不能保证正确的查询执行。我们引入了QO-Bench，一个用于类型化事件元组上查询操作符问答的诊断基准。该基准涵盖22,984篇新闻文章和614个公司事件，涉及18个查询模板，在785个问题上进行评估。每个黄金答案由类型化事件元组确定性计算得出，并通过召回率评分，答案通过精确匹配而非LLM评判器与黄金元组匹配。这种设计支持操作符级别的诊断，如连接和交集。我们在匹配条件下评估了RAG、ReAct RAG、GraphRAG和信息提取到SQL的方法，并设置了一个长上下文oracle上限以隔离检索失败。一个双轴框架——索引时保持与查询时执行——预测了每种范式失败的位置，结果证实了这一点：系统检索到相关文本，但丢弃了操作符所需的类型化值，并且可部署的范式排名在不同操作符间反转，相似性检索在过滤/投影上领先，而提取到SQL在交集和计数上领先。即使提供了黄金证据，长上下文oracle也远未饱和，因此操作符执行——而不仅仅是检索——是一个核心瓶颈，更强的答案模型也无法消除。QO-Bench将目标从段落相关性重新定义为查询操作符保持检索。
英文摘要：
Many real-world questions over business, legal, and scientific corpora are natural-language versions of database-style queries over records latent in text. Existing retrieval-augmented generation (RAG) systems are optimized primarily for semantic relevance, but retrieving plausible passages does not guarantee correct query execution. We introduce QO-Bench, a diagnostic benchmark for query-operator question answering over typed event tuples. The benchmark covers 22,984 news articles and 614 corporate events, with 18 query templates instantiating 785 questions. Each gold answer is deterministically computed from typed event tuples and scored by recall, with answers matched to the gold tuples by exact match rather than an LLM judge. This design enables operator-level diagnosis such as joins and intersection. We evaluate RAG, ReAct RAG, GraphRAG, and information-extraction-to-SQL under matched conditions, with a long-context oracle ceiling to isolate retrieval failure. A two-axis framework -- index-time preservation versus query-time execution -- predicts where each paradigm fails, and the results bear it out: systems retrieve relevant text but discard the typed values operators need, and the deployable paradigm ranking inverts across operators, with similarity retrieval leading on filter/project and extraction-to-SQL on intersection and counting. Even given the gold evidence, a long-context oracle stays far from saturated, so operator execution -- not retrieval alone -- is a core bottleneck that a stronger answer model does not remove. QO-Bench reframes the goal from passage relevance to query-operator-preserving retrieval. The benchmark, predictions, and code are released at https://github.com/ZHANG-MENGAO/qo-bench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
