2D-TPE:二维位置编码增强大语言模型的表格理解能力
2D-TPE: Two-Dimensional Positional Encoding Enhances Table Understanding for Large Language Models
AI总结:
本文提出二维表格位置编码2D-TPE,通过让注意力头动态选择行/列等遍历顺序来保留表格空间结构,在五个基准上超越强基线并展现出更好的大表格可扩展性。
摘要:
表格在各个领域中普遍存在,用于简洁地表示结构化信息。赋予大语言模型(LLMs)对表格数据进行推理的能力是一个被积极探索的方向。然而,由于典型的大语言模型仅支持一维(1D)输入,现有方法通常将二维(2D)表格结构展平为标记序列,这可能严重破坏空间关系,并导致关键上下文信息的不可避免的损失。在本文中,我们首先通过两个精心设计的代理任务,实证展示了这种展平操作对LLMs捕捉表格空间信息性能的有害影响。随后,我们引入了一种简单而有效的位置编码方法,称为“2D-TPE”(二维表格位置编码),以应对这一挑战。2D-TPE使每个注意力头能够动态选择上下文中标记的排列顺序来关注它们,其中每种排列代表表格的一种不同遍历模式,例如按列或按行遍历。2D-TPE在保持计算效率的同时,有效降低了丢失关键空间信息的风险,从而更好地保留表格结构。在五个基准上的大量实验表明,2D-TPE优于强基线,强调了保留表格结构对于准确表格理解的重要性。综合分析进一步揭示了2D-TPE相比基线在大表格上具有显著更好的可扩展性。
英文摘要:
Tables are ubiquitous across various domains for concisely representing structured information. Empowering large language models (LLMs) to reason over tabular data represents an actively explored direction. However, since typical LLMs only support one-dimensional~(1D) inputs, existing methods often flatten the two-dimensional~(2D) table structure into a sequence of tokens, which can severely disrupt the spatial relationships and result in an inevitable loss of vital contextual information. In this paper, we first empirically demonstrate the detrimental impact of such flattening operations on the performance of LLMs in capturing the spatial information of tables through two elaborate proxy tasks. Subsequently, we introduce a simple yet effective positional encoding method, termed ``2D-TPE'' (Two-Dimensional Table Positional Encoding), to address this challenge. 2D-TPE enables each attention head to dynamically select a permutation order of tokens within the context for attending to them, where each permutation represents a distinct traversal mode for the table, such as column-wise or row-wise traversal. 2D-TPE effectively mitigates the risk of losing essential spatial information while preserving computational efficiency, thus better preserving the table structure. Extensive experiments across five benchmarks demonstrate that 2D-TPE outperforms strong baselines, underscoring the importance of preserving the table structure for accurate table comprehension. Comprehensive analysis further reveals the substantially better scalability of 2D-TPE to large tables than baselines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。