arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.03402cs.CLcs.IR
收藏

ULLME:面向大语言模型嵌入的生成增强学习统一框架

ULLME: A Unified Framework for Large Language Model Embeddings with Generation-Augmented Learning

University of Oregon(俄勒冈大学);Adobe Research(奥多比研究院)

Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Thien Huu Nguyen

AI总结:

本文提出统一框架ULLME和生成增强表示学习方法GRL,通过双向注意力与生成一致性约束提升大语言模型的文本嵌入性能,并发布多个预训练模型在基准上取得优异表现。

摘要:

大语言模型(LLMs)在各种自然语言处理任务中表现出色,但利用它们进行稠密段落嵌入仍然具有挑战性。这源于其因果注意力机制以及预训练目标与文本排序任务之间的错位。尽管近期已有一些努力尝试解决这些问题,但现有的基于LLM的文本嵌入框架仅支持有限的LLM架构和微调策略,限制了其实际应用和通用性。在这项工作中,我们提出了大语言模型嵌入统一框架(ULLME),这是一个灵活、即插即用的实现,能够在各种LLM上实现双向注意力,并支持多种微调策略。我们还提出了生成增强表示学习(GRL),这是一种新颖的微调方法,用于提升LLM在文本嵌入任务上的表现。GRL强制基于表示的相似度分数与基于生成的相似度分数之间保持一致,利用LLM强大的生成能力来学习段落嵌入。为了展示我们框架的灵活性和有效性,我们发布了来自ULLME的三个预训练模型,它们具有不同的骨干架构,参数规模从1.5B到8B不等,所有模型都在Massive Text Embedding Benchmark上展现出强大的性能。我们的框架已公开发布于:https://github.com/nlp-uoregon/ullme。ULLME的演示视频也可在https://rb.gy/ws1ile找到。

英文摘要:

Large Language Models (LLMs) excel in various natural language processing tasks, but leveraging them for dense passage embedding remains challenging. This is due to their causal attention mechanism and the misalignment between their pre-training objectives and the text ranking tasks. Despite some recent efforts to address these issues, existing frameworks for LLM-based text embeddings have been limited by their support for only a limited range of LLM architectures and fine-tuning strategies, limiting their practical application and versatility. In this work, we introduce the Unified framework for Large Language Model Embedding (ULLME), a flexible, plug-and-play implementation that enables bidirectional attention across various LLMs and supports a range of fine-tuning strategies. We also propose Generation-augmented Representation Learning (GRL), a novel fine-tuning method to boost LLMs for text embedding tasks. GRL enforces consistency between representation-based and generation-based relevance scores, leveraging LLMs' powerful generative abilities for learning passage embeddings. To showcase our framework's flexibility and effectiveness, we release three pre-trained models from ULLME with different backbone architectures, ranging from 1.5B to 8B parameters, all of which demonstrate strong performance on the Massive Text Embedding Benchmark. Our framework is publicly available at: https://github.com/nlp-uoregon/ullme. A demo video for ULLME can also be found at https://rb.gy/ws1ile.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑