arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.09328cs.LGcs.AI
收藏

TransTab:学习跨表可转换的表格式Transformers

TransTab: Learning Transferable Tabular Transformers Across Tables

UIUC(伊利诺伊大学厄巴纳-香槟分校)

Zifeng Wang, Jimeng Sun

摘要:

表格数据(或表格)是机器学习(ML)中使用最广泛的数据格式。然而,ML模型通常假设表结构在训练和测试中保持固定。在ML建模之前,需要进行大量的数据清理,以合并具有不同列的不同表。这种预处理通常会导致显著的数据浪费(例如,移除不匹配的柱和样品)。如何从具有部分重叠列的多个表中学习ML模型?如何随着时间的推移随着更多列变得可用而增量地更新ML模型?我们可以在多个不同的表上利用模型预训练吗?如何训练一个可以在看不见的表上进行预测的ML模型?为了回答所有这些问题,我们建议放宽固定的表结构,引入一个可转移的表格Transformer(transTab)的表。TransTab的目标是将每个样本(表中的一行)转换为可泛化的嵌入向量,然后应用堆栈Transformers进行特征编码。一种方法学见解是将列描述和表单元格组合起来作为门控Transformer模型的原始输入。另一个观点是引入监督和自我监督的预训练来提高模型性能。我们在不同的基准数据集和五个肿瘤学临床试验数据集上比较了TransTab与多种基线方法。总体而言,在监督学习、特征增量学习和迁移学习场景中,TransTab分别在12种方法中排名1.00、1.00、1.78;并且所提出的预训练平均比监督学习提高了2.3%的AUC。

英文摘要:

Tabular data (or tables) are the most widely used data format in machine learning (ML). However, ML models often assume the table structure keeps fixed in training and testing. Before ML modeling, heavy data cleaning is required to merge disparate tables with different columns. This preprocessing often incurs significant data waste (e.g., removing unmatched columns and samples). How to learn ML models from multiple tables with partially overlapping columns? How to incrementally update ML models as more columns become available over time? Can we leverage model pretraining on multiple distinct tables? How to train an ML model which can predict on an unseen table? To answer all those questions, we propose to relax fixed table structures by introducing a Transferable Tabular Transformer (TransTab) for tables. The goal of TransTab is to convert each sample (a row in the table) to a generalizable embedding vector, and then apply stacked transformers for feature encoding. One methodology insight is combining column description and table cells as the raw input to a gated transformer model. The other insight is to introduce supervised and self-supervised pretraining to improve model performance. We compare TransTab with multiple baseline methods on diverse benchmark datasets and five oncology clinical trial datasets. Overall, TransTab ranks 1.00, 1.00, 1.78 out of 12 methods in supervised learning, feature incremental learning, and transfer learning scenarios, respectively; and the proposed pretraining leads to 2.3% AUC lift on average over the supervised learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑