arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.04113cs.CL
收藏

揭示大语言模型在表格信息查询中的局限性

Uncovering Limitations of Large Language Models in Information Seeking from Tables

Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS)(中国科学院智能信息处理重点实验室);Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Harbin Engineering University(哈尔滨工程大学);Peng Cheng Laboratory(鹏城实验室)

Chaoxu Pang, Yixuan Cao, Chunhao Yang, Ping Luo

AI总结:

本文提出更可靠的表格信息查询基准TabIS,采用单选题形式评估12个大语言模型,发现除GPT-4-turbo外多数模型表现不佳,并揭示其在表格结构理解与鲁棒性方面的局限。

摘要:

表格因其高信息密度和广泛使用而被认为是重要的信息来源。从表格中查询信息(TIS)是大语言模型(LLMs)的一项关键能力,也是基于知识的问答系统的基础。然而,该领域目前缺乏全面且可靠的评估。本文提出了一个更可靠的表格信息查询基准(TabIS)。为了避免基于文本相似度的评估指标带来的不可靠性,TabIS采用单选题形式(每题两个选项),而非文本生成形式。我们建立了一套有效的选项生成流程,以确保选项的难度和质量。在12个大语言模型上进行的实验表明,尽管GPT-4-turbo的表现勉强令人满意,但其他闭源模型和开源模型的表现均不理想。进一步分析显示,大语言模型对表格结构的理解较差,并且难以在TIS性能与对伪相关表格(常见于检索增强系统中)的鲁棒性之间取得平衡。这些发现揭示了大语言模型在从表格中查询信息方面的局限性和潜在挑战。我们公开了数据和代码,以促进该领域的进一步研究。

英文摘要:

Tables are recognized for their high information density and widespread usage, serving as essential sources of information. Seeking information from tables (TIS) is a crucial capability for Large Language Models (LLMs), serving as the foundation of knowledge-based Q&A systems. However, this field presently suffers from an absence of thorough and reliable evaluation. This paper introduces a more reliable benchmark for Table Information Seeking (TabIS). To avoid the unreliable evaluation caused by text similarity-based metrics, TabIS adopts a single-choice question format (with two options per question) instead of a text generation format. We establish an effective pipeline for generating options, ensuring their difficulty and quality. Experiments conducted on 12 LLMs reveal that while the performance of GPT-4-turbo is marginally satisfactory, both other proprietary and open-source models perform inadequately. Further analysis shows that LLMs exhibit a poor understanding of table structures, and struggle to balance between TIS performance and robustness against pseudo-relevant tables (common in retrieval-augmented systems). These findings uncover the limitations and potential challenges of LLMs in seeking information from tables. We release our data and code to facilitate further research in this field.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑