在多样化表格数据任务上测试统一序列到序列大语言模型预训练的极限
Testing the Limits of Unified Sequence to Sequence LLM Pretraining on Diverse Table Data Tasks
AI总结：
本研究针对多样化表格任务，探索基于编码器-解码器风格LLM的统一预训练方法，通过对不同规模T5模型的实验验证了自监督预训练的性能提升效果，首次系统研究了该统一方法跨规模的优势及指令微调变体的表现。
摘要：
存储在数据库中的表格以及网页、文章中出现的表格，是互联网上可用半结构化数据的重要组成部分。因此，开发一种基于大语言模型（LLM）的建模方法以解决语义解析、问答以及分类问题等多样化表格任务变得十分必要。传统上，每个任务都有各自独立的专用模型，这引出了一个问题：我们能在多大程度上构建一个统一模型，使其在部分表格任务上表现良好的同时，不会在其他任务上出现显著性能下降。为此，我们尝试在预训练阶段采用编码器-解码器风格的LLM构建一种共享建模方法，以适配多样化任务。我们使用表格数据及周边上下文对T5的不同模型族进行持续预训练和微调，并在不同模型规模下针对这些下游任务评估我们的方法。通过多项消融研究，我们发现采用自监督目标的预训练能够显著提升模型在这些任务上的性能。作为性能提升的一个例证，我们发现那些经过指令微调、专门用于文本问答（QA）且已在表格数据上训练过的公开模型，在表格特定问答任务上仍有提升空间。我们的工作首次研究了表格专用预训练统一方法在从770M到11B的序列到序列模型上扩展的优势，同时还对比了模型的指令微调变体。
英文摘要：
Tables stored in databases and tables which are present in web pages and articles account for a large part of semi-structured data that is available on the internet. It then becomes pertinent to develop a modeling approach with large language models (LLMs) that can be used to solve diverse table tasks such as semantic parsing, question answering as well as classification problems. Traditionally, there existed separate models specialized for each task individually. It raises the question of how far can we go to build a unified model that works well on some table tasks without significant degradation on others. To that end, we attempt at creating a shared modeling approach in the pretraining stage with encoder-decoder style LLMs that can cater to diverse tasks. We evaluate our approach that continually pretrains and finetunes different model families of T5 with data from tables and surrounding context, on these downstream tasks at different model scales. Through multiple ablation studies, we observe that our pretraining with self-supervised objectives can significantly boost the performance of the models on these tasks. As an example of one improvement, we observe that the instruction finetuned public models which come specialized on text question answering (QA) and have been trained on table data still have room for improvement when it comes to table specific QA. Our work is the first attempt at studying the advantages of a unified approach to table specific pretraining when scaled from 770M to 11B sequence to sequence models while also comparing the instruction finetuned variants of the models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
