RelationalFactQA：评估大型语言模型表格事实检索的基准
RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models
AI总结：
本文提出RelationalFactQA基准，用于评估LLMs从参数化知识生成结构化表格输出的能力，发现现有模型在关系型事实检索上准确率低于25%且随输出维度增加而下降。
摘要：
大型语言模型（LLMs）的事实性是一个持续的挑战。现有基准通常评估简短的事实性答案，而忽视了从参数化知识中生成结构化、多记录表格输出的关键能力。我们证明，这种关系型事实检索比孤立的点式查询要困难得多，即使模型已知单个事实，也会暴露出对输出维度（例如属性或记录的数量）敏感的独特失败模式。为了系统地评估这一探索不足的能力，我们引入了RelationalFactQA，这是一个新的基准，包含多样化的自然语言问题（配对SQL）和金标准表格答案，专门设计用于评估结构化格式的知识检索。RelationalFactQA支持跨不同查询复杂度、输出规模和数据特征的分析。我们的实验表明，即使是最先进的LLMs也面临显著困难，在生成关系型输出时事实准确率不超过25%，且性能随着输出维度的增加而显著下降。这些发现突显了当前LLMs在综合结构化事实知识方面的关键局限性，并确立了RelationalFactQA作为衡量LLM事实性未来进展的重要资源。
英文摘要：
Factuality in Large Language Models (LLMs) is a persistent challenge. Current benchmarks often assess short factual answers, overlooking the critical ability to generate structured, multi-record tabular outputs from parametric knowledge. We demonstrate that this relational fact retrieval is substantially more difficult than isolated point-wise queries, even when individual facts are known to the model, exposing distinct failure modes sensitive to output dimensionality (e.g., number of attributes or records). To systematically evaluate this under-explored capability, we introduce RelationalFactQA, a new benchmark featuring diverse natural language questions (paired with SQL) and gold-standard tabular answers, specifically designed to assess knowledge retrieval in a structured format. RelationalFactQA enables analysis across varying query complexities, output sizes, and data characteristics. Our experiments reveal that even state-of-the-art LLMs struggle significantly, not exceeding 25% factual accuracy in generating relational outputs, with performance notably degrading as output dimensionality increases. These findings underscore critical limitations in current LLMs' ability to synthesize structured factual knowledge and establish RelationalFactQA as a crucial resource for measuring future progress in LLM factuality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
