大语言模型（LLM）能否使用关系Transformer嵌入？
Can LLMs Use Relational Transformer Embeddings?
浏览论文内容
中文总结 AI 辅助
本研究探索将关系Transformer嵌入作为软令牌注入大语言模型的可行性，经多任务评估发现该混合模型表现不佳，需更强对齐目标与感知模式设计才能成为关系预测可靠途径。
中文摘要 AI 辅助
将冻结的关系编码器嵌入作为软令牌注入大语言模型（LLM）是一种概念上有吸引力的融合策略：编码器处理多表结构，LLM处理语言与推理，且无需有损文本序列化。我们通过将冻结的关系Transformer（RT）的嵌入，经学习得到的MLP投影和LoRA适配注入Qwen3.5-4B，先在思维链推理轨迹上进行监督微调（SFT）训练，再进行基于组的强化学习（GSPO）训练，对RelBench中6个关系数据库的10个二分类任务，在单任务（ST）、数据集内（WD）、跨数据集（CD）和全任务（ALL）4种监督机制下进行评估。该混合模型未始终优于独立RT：其表现常低于随机水平，对序列化格式和关系令牌预算高度敏感，在RL训练下不稳定。我们报告这些负面结果并分析失败模式，认为软令牌融合需更强的对齐目标和感知模式的设计，才能成为关系预测的可靠途径。
英文摘要
Injecting frozen relational-encoder embeddings as soft tokens into a large language model (LLM) is a conceptually appealing fusion strategy: the encoder handles multi-table structure, the LLM handles language and reasoning, and no lossy text serialization is required. We test this hypothesis concretely by injecting embeddings from a frozen Relational Transformer (RT) into Qwen3.5-4B via a learned MLP projection and LoRA adaptation, trained first with supervised fine-tuning (SFT) on chain-of-thought reasoning traces and then with group-based reinforcement learning (GSPO). We evaluate across 10 binary classification tasks on 6 relational databases from RelBench, under four supervision regimes: single-task (ST), within-dataset (WD), cross-dataset (CD), and all-task (ALL). The hybrid model does not consistently outperform standalone RT: it is frequently below random, highly sensitive to serialization format and relational-token budget, and unstable under RL training. We report these negative results and analyze the failure modes, arguing that soft-token fusion requires stronger alignment objectives and schema-aware design before it can serve as a reliable route to relational prediction.
发表机构
- Kunumi Institute（库鲁米研究所）
- Universidade Federal de Minas Gerais（米纳斯吉拉斯联邦大学）
机构由 AI 辅助整理，请以论文原文为准。
