arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.23073cs.LGcs.AI
收藏

通过学习的潜在多表表示构建先验数据适配的关系模型的路径

Path to Prior-Data Fitted Relational Models via Learned Latent Multi-table Representations

SAP Labs(思爱普实验室)

Mohammad Sadeq Abolhasani, Viswanath Ganapathy

AI总结:

该研究在合成关系数据库上预训练Griffin模型,采用特定合成课程学习策略,将其潜在表示传入冻结的\ticl{},在16个关系分类任务上取得优于基线的性能,证明PFN式预训练可迁移至图关系基础模型。

摘要:

先验数据适配网络(PFNs)表明,在大量合成任务组成的群体上进行预训练,可实现对未见数据集的泛化,仅需少量或无需下游微调。该范式对基于Transformer的表格和关系基础模型(如RDB-PFN)有效,但对于基于消息传递的图关系模型,该范式仍未得到充分探索。我们在大量合成关系数据库集合上预训练基于图的关系基础模型Griffin,探究对Transformer式关系学习器有益的课程学习是否也能改进消息传递表示。我们未对下游任务进行微调,而是将预训练编码器用作通用关系表示学习器,将其潜在表示传递给冻结的上下文内预测器(\ticl{}),所有组件均未在目标任务上进行训练。在16个真实世界关系分类任务上,针对原始特征和未训练编码器基线,以5个随机种子评估,从模式引导数据开始、接着是模式无关数据的纯合成课程,实现了最强的冻结上下文内性能,优于仅真实数据预训练及所有语料库的无序单阶段混合;在真实数据预训练后追加该课程,可得到整体最佳编码器。这些结果为PFN式合成预训练可迁移至基于图的关系基础模型提供了初步证据。

英文摘要:

Prior-data fitted networks (PFNs) show that pretraining on large populations of synthetic tasks can enable generalization to unseen datasets with little or no downstream fine-tuning. This paradigm is effective for transformer-based tabular and relational foundation models such as RDB-PFN, but remains largely unexplored for graph-based relational models built on message passing. We pretrain the graph-based relational foundation model Griffin on large collections of synthetic relational databases and ask whether curriculum learning, which benefits transformer-based relational learners, also improves message-passing representations. Rather than fine-tune on downstream tasks, we use the pretrained encoder as a general-purpose relational representation learner and pass its latent representations to a frozen in-context predictor (\ticl{}); no component is trained on the target task. On 16 real-world relational classification tasks, evaluated over five seeds against raw-feature and untrained-encoder baselines, a purely synthetic curriculum that begins with schema-guided data and continues with schema-agnostic data gives the strongest frozen in-context performance\rev{, outperforming real-data-only pretraining and every unordered single-stage mixture of the corpora; when appended after real-data pretraining, it yields the best encoders overall}. These results provide initial evidence that PFN-style synthetic pretraining transfers to graph-based relational foundation models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑