SciTaRC：基于科学表格数据的问答基准测试，需语言推理与复杂计算
SciTaRC: A Plan-Annotated Scientific Tabular QA Benchmark for Language Reasoning and Complex Computation
AI总结：
SciTaRC是一个针对科学表格数据的问答基准测试，揭示了现有AI模型在语言推理和复杂计算任务上的不足。
摘要：
我们介绍了SciTaRC，一个由专家编写的基准测试，用于测试关于科学论文中表格数据的问题，这些问题需要深入的语言推理和复杂计算。我们显示，当前最先进的AI模型在至少23%的问题上失败，即使对于能力强大的开放式权重模型如Llama-3.3-70B-Instruct，也有65.5%的任务失败。我们的分析揭示了一个普遍的'执行瓶颈'：无论是代码还是语言模型，在提供正确策略的情况下，都难以忠实执行计划。具体来说，基于代码的方法在原始科学表格上表现脆弱，而自然语言推理主要由于初始理解问题和计算错误而失败。
英文摘要：
We introduce SciTaRC, an expert-authored benchmark for question answering over scientific tables that targets composite, multi-step reasoning. To enable fine-grained diagnostic analysis beyond end-task accuracy, SciTaRC pairs each question with a manually constructed reasoning plan and explicit complexity metrics. State-of-the-art models fail on at least 23% of these questions, while highly capable open-weight models like Llama-3.3-70B collapse on 65.5% of the benchmark. Error analysis shows that, in zero-shot settings, failures are driven primarily by question comprehension, where models misinterpret the scientific query and derive the wrong reasoning objective. To determine whether overcoming this gap is sufficient, we use the structured plans to decouple strategy formulation from execution. Surprisingly, providing oracle step-by-step plans yields only limited gains and fails to eliminate the performance gap. This reveals a substantial execution bottleneck: both natural language and code-based methods struggle to reliably carry out long-horizon computational chains over structured data. Ultimately, SciTaRC serves as a rigorous diagnostic testbed for studying both planning and execution in scientific table reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
