语言和视觉领域的基础模型之所以能够对任意文本和视觉输入执行推理,得益于可迁移表示,例如语言中的 token 词汇表。知识图谱(KG)具有通常互不重叠的不同实体和关系词汇表。在 KG 上设计基础模型的关键挑战,是学习这种可迁移表示,从而能够在具有任意实体和关系词汇表的任意图上进行推理。在本工作中,我们朝着此类基础模型迈出一步,并提出 ULTRA,这是一种用于学习通用且可迁移的图表示的方法。ULTRA 将关系表示构建为以关系之间的交互为条件的函数。这种条件化策略使预训练的 ULTRA 模型能够归纳泛化到具有任意关系词汇表的任何未见 KG,并可在任意图上进行微调。通过在 57 个不同 KG 上开展链接预测实验,我们发现,单个预训练 ULTRA 模型在各种规模的未见图上的零样本归纳推理性能,往往与在特定图上训练的强基线相当,甚至更优。微调还会进一步提升性能。
英文摘要:
Foundation models in language and vision have the ability to run inference on any textual and visual inputs thanks to the transferable representations such as a vocabulary of tokens in language. Knowledge graphs (KGs) have different entity and relation vocabularies that generally do not overlap. The key challenge of designing foundation models on KGs is to learn such transferable representations that enable inference on any graph with arbitrary entity and relation vocabularies. In this work, we make a step towards such foundation models and present ULTRA, an approach for learning universal and transferable graph representations. ULTRA builds relational representations as a function conditioned on their interactions. Such a conditioning strategy allows a pre-trained ULTRA model to inductively generalize to any unseen KG with any relation vocabulary and to be fine-tuned on any graph. Conducting link prediction experiments on 57 different KGs, we find that the zero-shot inductive inference performance of a single pre-trained ULTRA model on unseen graphs of various sizes is often on par or better than strong baselines trained on specific graphs. Fine-tuning further boosts the performance.
Expanding Reasoning Potential in Foundation Model by Learning Diverse Chains of Thought Patterns
School of Computer Science, Peking University(北京大学计算机科学系);Meituan(美团);State Key Laboratory of Multimedia Information Processing, Peking University(北京大学多媒体信息处理国家重点实验室)
A Prompt-Based Knowledge Graph Foundation Model for Universal In-Context Reasoning
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);National Institute of Healthcare Data Science, Nanjing University(南京大学国家卫生数据科学研究院)
Beyond Completion: A Foundation Model for General Knowledge Graph Reasoning
Zhejiang University(浙江大学);Shopee Pte.Ltd.(Shopee公司);University of Macau(澳门大学);Zhejiang Key Laboratory of Big Data Intelligent Computing(浙江大学大数据智能计算重点实验室)
Knowledge Reasoning Language Model: Unifying Knowledge and Language for Inductive Knowledge Graph Reasoning
The Key Laboratory of Knowledge Engineering with Big Data (the Ministry of Education of China), Hefei University of Technology, China(合肥工业大学大数据知识工程教育部重点实验室);School of Computer Science and Information Engineering, Hefei University of Technology, China(合肥工业大学计算机与信息学院);Nanjing University of Science and Technology, China(南京理工大学);China Unicom Digital Technology Co., Ltd., Beijing, China(联通数字科技有限公司);China Unicom Internet of Things Co., Ltd., Nanjing, China(联通物联网有限责任公司);Shandong Inspur Science Research Institute, Jinan, China(山东浪潮科学研究院);Griffith University, Australia(格里菲斯大学)
Benchmarking Foundation Models with Retrieval-Augmented Generation in Olympic-Level Physics Problem Solving
AAII, University of Technology Sydney, New South Wales, Australia(AAII,悉尼大学,新南威尔士州,澳大利亚);Eindhoven University of Technology, Eindhoven, The Netherlands(埃因霍温理工大学,埃因霍温,荷兰);University of Liverpool, Liverpool, United Kingdom(利物浦大学,利物浦,英国);University of New South Wales, New South Wales, Australia(新南威尔士大学,新南威尔士州,澳大利亚)
SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning
Nanyang Technological University(南洋理工大学);Institute for Infocomm Research (IR), A*STAR(新加坡科技研究局信息通信研究所);Centre for Frontier AI Research (CFAR), A*STAR(新加坡科技研究局前沿人工智能研究中心)
LINKED: Eliciting, Filtering and Integrating Knowledge in Large Language Model for Commonsense Reasoning
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学);Shanghai AI Lab(上海人工智能实验室);Dalian University of Technology(大连理工大学);Fudan University(复旦大学);Hong Kong University of Science and Technology(香港科技大学);Peking University(北京大学)
Knowledge Crosswords: Geometric Knowledge Reasoning with Large Language Models
The Hong Kong University of Science and Technology(香港科技大学);University of Washington(华盛顿大学);Xi’an Jiaotong University(西安交通大学);University of Notre Dame(圣母大学);Carnegie Mellon University(卡内基梅隆大学)
MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models
Department of Computer Science and Engineering Bangladesh University of Engineering and Technology (BUET)(计算机科学与工程系,孟加拉国工程与技术大学);Statistics, Islamic University, Bangladesh(统计学,伊斯兰大学,孟加拉国);Faculty of Information Technology, Monash University, Melbourne, Australia(信息科技学院,墨尔本大学,澳大利亚);Qatar Computing Research Institute (QCRI)(卡塔尔计算研究所)
LegalOne: A Family of Foundation Models for Reliable Legal Reasoning
Department of Computer Science, Tsinghua University(清华大学计算机科学系);Quancheng Laboratory(清华云城实验室);University of Waterloo, Canada(加拿大滑铁卢大学);China University of Political Science and Law(中国政法大学);MegaTech.AI Inc(MegaTech.AI公司)
Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning
Department of Computer Science, Yale University;Department of Molecular \& Cellular Biology, Harvard University;Allen Institute for AI;Department of Computer Science, Northwestern University
KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graph
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);Neusoft AI Magic Technology Research(东软AI魔法技术研究院);Neusoft Institute of Intelligent Medical Research(东软智能医学研究院)