arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.11235cs.CL
收藏

GT2Vec:大型语言模型作为文本和图结构数据的多模态编码器

GT2Vec: Large Language Models as Multi-Modal Encoders for Text and Graph-Structured Data

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Amazon(亚马逊公司);Michigan State University(密歇根州立大学);University of Washington(华盛顿大学);University of Michigan(密歇根大学)

Jiacheng Lin, Kun Qian, Haoyu Han, Nurendra Choudhary, Tianxin Wei, Zhongruo Wang, Sahika Genc, Edward W Huang, Sheng Wang, Karthik Subbian, Danai Koutra, Jimeng Sun

AI总结:

针对现有方法难以充分利用图与文本模态异质性的问题,提出GT2Vec框架,通过MLP适配器和对比学习利用大型语言模型联合编码文本与图数据,在三类任务的六个数据集上持续优于基线,有效提升联合嵌入质量。

摘要:

图结构信息提供了丰富的上下文信息,可通过提供结构化关系和层次结构来增强语言模型,从而为检索、问答和分类等各种应用生成更具表现力的嵌入。然而,现有的基于多层感知机(MLPs)或浅层Transformer的图与文本嵌入集成方法,在充分利用这些模态异质性方面能力有限。为解决此问题,我们提出GT2Vec——一个利用大型语言模型(LLMs)联合编码文本和图数据的简单且有效的框架。具体而言,GT2Vec采用MLP适配器将图嵌入投影到与文本嵌入相同的空间,使LLM能联合处理两种模态;与先前工作不同,我们还引入对比学习以更有效地对齐图和文本空间,从而提升学习到的联合嵌入质量。在涵盖知识图上下文问答、图-文本对分类和检索三个任务的六个数据集上的实证结果表明,GT2Vec持续优于现有基线,在多个数据集上取得显著改进;消融研究进一步验证了我们方法的有效性。

英文摘要:

Graph-structured information offers rich contextual information that can enhance language models by providing structured relationships and hierarchies, leading to more expressive embeddings for various applications such as retrieval, question answering, and classification. However, existing methods for integrating graph and text embeddings, often based on Multi-layer Perceptrons (MLPs) or shallow transformers, are limited in their ability to fully exploit the heterogeneous nature of these modalities. To overcome this, we propose GT2Vec, a simple yet effective framework that leverages Large Language Models (LLMs) to jointly encode text and graph data. Specifically, GT2Vec employs an MLP adapter to project graph embeddings into the same space as text embeddings, allowing the LLM to process both modalities jointly. Unlike prior work, we also introduce contrastive learning to align the graph and text spaces more effectively, thereby improving the quality of learned joint embeddings. Empirical results across six datasets spanning three tasks, knowledge graph-contextualized question answering, graph-text pair classification, and retrieval, demonstrate that GT2Vec consistently outperforms existing baselines, achieving significant improvements across multiple datasets. These results highlight GT2Vec's effectiveness in integrating graph and text data. Ablation studies further validate the effectiveness of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑