TabularMath: 通过大规模语言模型理解表格上的数学推理
TabularMath: Understanding Math Reasoning over Tables with Large Language Models
AI总结：
本文提出TabularMath基准，通过神经符号框架将数学问题转化为可扩展的表格推理任务，揭示表格复杂度、质量及表征对推理性能的影响，发现低质量表格对当前LLM的可靠性构成严重风险。
摘要：
数学推理长期以来一直是评估大规模语言模型的关键基准。尽管在数学文字问题上取得了显著进展，但现实应用中对表格数据的推理需求却被忽视。例如，商业智能应用不仅需要多步数值推理，还要求对不完整或不一致信息的鲁棒性。然而，该领域的全面评估严重受限，受限于手动收集的表格难以扩展，且缺乏对现实场景中潜在陷阱的覆盖。为解决此问题，我们提出AutoT2T，一种神经符号框架，能够可控地将数学文字问题转化为可扩展且验证的表格推理任务。基于此流程，我们开发了包含四个子集的TabularMath基准，涵盖文本和图像表格，覆盖表格复杂性、质量及表征维度。我们的研究揭示了三个关键发现：（1）表格复杂度和推理难度共同影响推理性能；（2）低质量表格对当前LLM的可靠推理构成严重风险；（3）不同表格模态显示相似趋势，文本表格通常比图像表格更容易被模型推理。对每个发现进行了深入分析，以指导未来研究。
英文摘要：
Mathematical reasoning has long been a key benchmark for evaluating large language models. Although substantial progress has been made on math word problems, the need for reasoning over tabular data in real-world applications has been overlooked. For instance, applications such as business intelligence demand not only multi-step numerical reasoning with tables but also robustness to incomplete or inconsistent information. However, comprehensive evaluation in this area is severely limited, constrained by the reliance on manually collected tables that are difficult to scale and the lack of coverage for potential traps encountered in real-world scenarios. To address this problem, we propose AutoT2T, a neuro-symbolic framework that controllably transforms math word problems into scalable and verified tabular reasoning tasks. Building on this pipeline, we develop TabularMath, a benchmark comprising four subsets that include both text-based and image-based tables, covering table complexity, table quality, and table representation dimensions. Our study reveals three key observations: (1) Table complexity and reasoning difficulty impact reasoning performance jointly; (2) Low-quality tables pose severe risks to reliable reasoning in current LLMs; (3) Different table modalities show similar trends, with text-based tables typically being easier for models to reason over. In-depth analyses are conducted for each observation to guide future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
