用于不相交表格数据迁移学习的交叉注意力广义上下文
Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data
浏览论文内容
中文总结 AI 辅助
本文针对不相交表格数据迁移学习的共享特征假设不切实际的问题，提出CATTLE方法，通过Transformer投影权重捕获广义上下文实现跨域注意力迁移学习，在十组数据集上优于九种基线方法，取得平均排名2.9、平均AUROC提升3.7%的效果。
中文摘要 AI 辅助
与图像和文本不同，将迁移学习应用于表格数据颇具挑战，因为不同领域间的特征类型、结构和语义存在异质性。现有方法假设不同数据表间存在共享特征以实现领域间的知识迁移，而这在实际场景中并不现实。本文引入广义上下文学习，以消除对领域间共享特征的要求。由Transformer投影权重捕获的针对键（key）、值（value）和查询（query）的广义上下文，提供了基于规则的泛化能力，而非传统上从Transformer激活中学习的特定领域上下文。源域的键投影权重与目标域的查询权重交互，以数据不可知的方式实现跨域注意力迁移学习（Cross-domain Attention Transfer Learning，CATTLE）。我们在十对不相交的源-目标数据集上开展实验，结果显示CATTLE可从单个源数据集中学习广义上下文，且在排名和统计性能上优于九种最先进的基线方法，包括机器学习、深度学习以及使用大规模预训练模型的迁移学习方法。CATTLE取得了最佳平均排名（2.9），并较基线方法实现了3.7%的平均AUROC提升。CATTLE的源代码可在该https URL获取。
英文摘要
Unlike images and text, applying transfer learning to tabular data is challenging due to heterogeneity in feature types, structures, and semantics across disparate domains. Existing methods assume shared features across data tables to enable knowledge transfer between domains, which is unrealistic in practice. \mds{This paper introduces generalized context learning to remove the requirement of shared features across domains. The generalized context captured by transformer projection weights for $key$, $value$, and $query$ provides rule-based generalization rather than the domain-specific context conventionally learned from transformer activations. Projection weights for $key$ from the source domain interact with the weight for $query$ in the target domain to achieve Cross-domain Attention Transfer Learning (CATTLE) in a data-agnostic manner. Our experiments on ten pairs of disjoint source-target data sets show that CATTLE can learn generalized context from a single source data set and is rank-wise and statistically superior to nine state-of-the-art baselines, including machine learning, deep learning, and transfer learning methods using large-scale pre-trained models. CATTLE achieves the best average rank (2.9) and delivers a 3.7% average AUROC gain over the baseline methods.} The CATTLE source code is available at https://tinyurl.com/pr5s8ywn.
发表机构
- Tennessee State University（田纳西州立大学）
- North Carolina Agricultural and Technical State University（北卡罗来纳农工州立大学）
机构由 AI 辅助整理，请以论文原文为准。
