通过检索增强的大语言模型实现表格数据的可扩展上下文学习
Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models
AI总结：
本文提出检索增强的大语言模型方法，通过定制检索模块与检索引导指令微调突破序列长度限制，实现表格数据上的可扩展上下文学习，在69个数据集上显著提升性能并展现独特优势。
摘要：
近期研究表明，大语言模型（LLMs）在针对表格数据进行后训练定制后，能够获得通用的表格上下文学习（TabICL）能力。这些模型能够跨不同数据模式和不同任务领域进行有效迁移。然而，现有基于LLM的TabICL方法受限于少样本场景，原因在于LLM的序列长度限制，因为以纯文本表示的表格实例会消耗大量token。为了解决这一限制并实现任意数据规模下的可扩展TabICL，我们提出了针对表格数据定制的检索增强LLMs。我们的方法引入了一个定制化的检索模块，并结合面向LLM的检索引导指令微调。这使得LLMs能够有效利用更大规模的数据集，在69个广泛认可的数据集上取得了显著提升的性能，并展现出有前景的扩展行为。与最先进的表格模型的大量对比表明，尽管基于LLM的TabICL在整体性能上仍落后于经过良好调优的数值模型，但它在有限上下文条件下揭示了强大的算法，增强了集成多样性，并在特定数据集上表现优异。这些独特性质凸显了语言作为可扩展表格数据学习的通用且易用接口的潜力。
英文摘要：
Recent studies have shown that large language models (LLMs), when customized with post-training on tabular data, can acquire general tabular in-context learning (TabICL) capabilities. These models are able to transfer effectively across diverse data schemas and different task domains. However, existing LLM-based TabICL approaches are constrained to few-shot scenarios due to the sequence length limitations of LLMs, as tabular instances represented in plain text consume substantial tokens. To address this limitation and enable scalable TabICL for any data size, we propose retrieval-augmented LLMs tailored to tabular data. Our approach incorporates a customized retrieval module, combined with retrieval-guided instruction-tuning for LLMs. This enables LLMs to effectively leverage larger datasets, achieving significantly improved performance across 69 widely recognized datasets and demonstrating promising scaling behavior. Extensive comparisons with state-of-the-art tabular models reveal that, while LLM-based TabICL still lags behind well-tuned numeric models in overall performance, it uncovers powerful algorithms under limited contexts, enhances ensemble diversity, and excels on specific datasets. These unique properties underscore the potential of language as a universal and accessible interface for scalable tabular data learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
