arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.17867cs.LGcs.AIcs.DB
收藏

RT-J:面向上下文高效预测的关系Transformer大规模预训练

RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions

Stanford University(斯坦福大学);SAP(思爱普)

Rishabh Ranjan, Vignesh Kothapalli, Harshvardhan Agarwal, Charilaos Kanatsoulis, Roshan Upendra, Tom Palczewski, Carlos Guestrin, Jure Leskovec

AI总结:

本文提出85M参数的关系基础模型RT-J,通过构建含639个数据库的数据集、轻量级随机游走检索器及上下文调优等技术,在RelBench回归任务上仅需1/10示例即可达到同类模型性能,推进了关系数据上下文高效预测的前沿。

摘要:

面向表格和关系数据库的基础模型正越来越多地应用于各类科学与工业领域的预测任务。然而,将现有模型适配到未见过的数据集需要数万条示例。本文提出RT-J,这是一个拥有85M参数的关系基础模型(RFM),即便仅使用数百条或更少的上下文示例也能做出准确预测。为实现这一目标,我们首先构建了Join,这是一个包含639个网页爬取数据库、6023个由大语言模型(LLM)生成的预测任务的大型集合。我们使用最长达8192个数据库单元的长上下文进行预训练,采用来自新型轻量级随机游走检索器的上下文示例。最后,我们利用上下文调优和集成来扩展测试时的计算资源。在RelBench的上下文回归任务上,RT-J平均仅需1/10的示例,即可达到使用LLM智能体或RDBLearn特征化器以及TabICLv2或LightGBM预测器的强大特征工程流水线的性能。总体而言,本研究推进了关系数据上下文高效预测的前沿,并强调了现实语义、上下文工程以及关系基础模型(RFM)大规模预训练的重要性。

英文摘要:

Foundation models for tables and relational databases are increasingly used for predictive tasks across various scientific and industrial domains. However, adapting existing models to unseen datasets requires tens of thousands of examples. Here we introduce RT-J, an 86M-parameter relational foundation model (RFM) that makes accurate predictions even with hundreds of in-context examples or fewer. To achieve this, we first assemble the Join, a large collection of 639 web-crawled databases with 6,023 LLM-generated forecasting tasks. We pretrain with long contexts of up to 8,192 database cells, using in-context examples from a novel lightweight random-walk retriever. Finally, we leverage context tuning and ensembling to scale up test-time compute. On in-context regression tasks from RelBench, RT-J requires 10x fewer examples on average to match strong feature engineering pipelines with LLM-agent or RDBLearn featurizers and TabICLv2 or LightGBM predictors. Overall, our work pushes the frontier for context-efficient predictions on relational data and highlights the importance of real-world semantics, context engineering and large-scale pretraining for RFMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑