RUST-BENCH：在结构化表格中的非结构化文本上基准测试大语言模型推理能力
RUST-BENCH: Benchmarking LLM Reasoning on Unstructured Text within Structured Tables
AI总结：
本文提出RUST-BENCH基准，基于2031张真实异构表格的7966个问题，从规模、异构性、领域专属性与推理复杂度联合评估LLMs，发现现有模型在多跳推理上仍存在明显不足。
摘要：
现有的表格推理基准大多在小型、格式统一的表格上测试模型，低估了真实世界数据的复杂性，也无法全面反映大语言模型（LLMs）的推理能力。真实表格往往篇幅长、结构异构且具有领域专属性，将结构化字段与自由文本混合在一起，并要求跨数千个token进行多跳推理。为弥补这一不足，我们提出了RUST-BENCH，一个包含2031张真实世界表格、共7966个问题的基准，覆盖两个领域：i) RB-Science（NSF基金记录）和ii) RB-Sports（NBA统计数据）。与以往工作不同，RUST-BENCH从规模、异构性、领域专属性以及推理复杂度四个维度联合评估LLMs。在开源和专有模型上的实验表明，LLMs在面对异构模式和复杂多跳推理时表现不佳，揭示了当前架构和提示策略中持续存在的弱点。RUST-BENCH为推进表格推理研究建立了一个具有挑战性的新测试平台。
英文摘要：
Existing tabular reasoning benchmarks mostly test models on small, uniform tables, underrepresenting the complexity of real-world data and giving an incomplete view of Large Language Models' (LLMs) reasoning abilities. Real tables are long, heterogeneous, and domain-specific, mixing structured fields with free text and requiring multi-hop reasoning across thousands of tokens. To address this gap, we introduce RUST-BENCH, a benchmark of 7966 questions from 2031 real-world tables spanning two domains: i) RB-Science (NSF grant records) and ii) RB-Sports (NBA statistics). Unlike prior work, RUST-BENCH evaluates LLMs jointly across scale, heterogeneity, domain specificity, and reasoning complexity. Experiments with open-source and proprietary models show that LLMs struggle with heterogeneous schemas and complex multi-hop inference, revealing persistent weaknesses in current architectures and prompting strategies. RUST-BENCH establishes a challenging new testbed for advancing tabular reasoning research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
