arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.14717cs.CL
收藏

表格LLM真正重要的是什么?模型与数据影响的元评估

What Really Matters for Table LLMs? A Meta-Evaluation of Model and Data Effects

University of Michigan(密歇根大学);AWS AI Labs(AWS人工智能实验室);Figma;OKX;Google(谷歌)

Naihao Deng, Sheng Zhang, Henghui Zhu, Shuaichen Chang, Jiani Zhang, Alexander Hanbo Li, Chung-Wei Hang, Hideo Kobayashi, Yiqun Hu, Patrick Ng

AI总结:

通过指令微调12个模型并在16个基准上评估,发现基座模型选择比训练数据对性能影响更大,泛化与推理仍是挑战。

摘要:

表格建模已经发展了数十年。在这项工作中,我们重新审视了这一轨迹,并强调了LLM时代出现的新挑战,特别是选择悖论:在表格指令微调的背景下,由于基础模型和训练集的多样性,难以将性能提升归因于特定因素。我们通过指令微调三个基础模型在四个现有数据集上,复制了四个表格LLM,共得到12个模型。然后我们在16个表格基准上评估这些模型。我们的研究首次定量分离了训练数据和基础模型选择的影响,揭示了基础模型选择比训练数据本身起更主导的作用。泛化和推理仍然具有挑战性,需要未来在表格建模上继续努力。基于我们的发现,我们分享了对表格建模未来方向的思考。

英文摘要:

Table modeling has progressed for decades. In this work, we revisit this trajectory and highlight emerging challenges in the LLM era, particularly the paradox of choice: the difficulty of attributing performance gains amid diverse base models and training sets in the context of table instruction tuning. We replicate four table LLMs by instruction-tuning three foundation models on four existing datasets, yielding 12 models. We then evaluate these models across 16 table benchmarks. Our study is the first to quantitatively disentangle the effects of training data and base model selection, revealing that base model choice plays a more dominant role than the training data itself. Generalization and reasoning remain challenging, inviting future effort on table modeling. Based on our findings, we share our thoughts on the future directions for table modeling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑