大语言模型(LLMs)在各种自然语言处理任务中表现出色,但利用它们进行稠密段落嵌入仍然具有挑战性。这源于其因果注意力机制以及预训练目标与文本排序任务之间的错位。尽管近期已有一些努力尝试解决这些问题,但现有的基于LLM的文本嵌入框架仅支持有限的LLM架构和微调策略,限制了其实际应用和通用性。在这项工作中,我们提出了大语言模型嵌入统一框架(ULLME),这是一个灵活、即插即用的实现,能够在各种LLM上实现双向注意力,并支持多种微调策略。我们还提出了生成增强表示学习(GRL),这是一种新颖的微调方法,用于提升LLM在文本嵌入任务上的表现。GRL强制基于表示的相似度分数与基于生成的相似度分数之间保持一致,利用LLM强大的生成能力来学习段落嵌入。为了展示我们框架的灵活性和有效性,我们发布了来自ULLME的三个预训练模型,它们具有不同的骨干架构,参数规模从1.5B到8B不等,所有模型都在Massive Text Embedding Benchmark上展现出强大的性能。我们的框架已公开发布于:https://github.com/nlp-uoregon/ullme。ULLME的演示视频也可在https://rb.gy/ws1ile找到。
英文摘要:
Large Language Models (LLMs) excel in various natural language processing tasks, but leveraging them for dense passage embedding remains challenging. This is due to their causal attention mechanism and the misalignment between their pre-training objectives and the text ranking tasks. Despite some recent efforts to address these issues, existing frameworks for LLM-based text embeddings have been limited by their support for only a limited range of LLM architectures and fine-tuning strategies, limiting their practical application and versatility. In this work, we introduce the Unified framework for Large Language Model Embedding (ULLME), a flexible, plug-and-play implementation that enables bidirectional attention across various LLMs and supports a range of fine-tuning strategies. We also propose Generation-augmented Representation Learning (GRL), a novel fine-tuning method to boost LLMs for text embedding tasks. GRL enforces consistency between representation-based and generation-based relevance scores, leveraging LLMs' powerful generative abilities for learning passage embeddings. To showcase our framework's flexibility and effectiveness, we release three pre-trained models from ULLME with different backbone architectures, ranging from 1.5B to 8B parameters, all of which demonstrate strong performance on the Massive Text Embedding Benchmark. Our framework is publicly available at: https://github.com/nlp-uoregon/ullme. A demo video for ULLME can also be found at https://rb.gy/ws1ile.
Embedding-Informed Adaptive Retrieval-Augmented Generation of Large Language Models
University of New South Wales(新南威尔士大学);University of California San Diego(加利福尼亚大学圣迭戈分校);Duke University(杜克大学);Georgia Institute of Technology(佐治亚理工学院);Adobe Research(奥多比研究院);CSIRO’s Data61(联邦科学与工业研究组织Data61)
Embedding Domain Knowledge for Large Language Models via Reinforcement Learning from Augmented Generation
Laboratory of Speech and Intelligent Information Processing, Institute of Acoustics, Chinese Academy of Sciences(语音与智能信息处理实验室,声学研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);China Southern Power Grid Artificial Intelligence Technology Co., Ltd.(中国南方电网人工智能技术有限公司)
Attention Expansion: Enhancing Keyphrase Extraction from Long Documents with Attention-Augmented Contextualized Embeddings
Institute for Research in Technology, ICAI School of Engineering, Comillas Pontifical University(技术研究所,ICAI工程学院,科米利亚斯宗座大学);DD-AIM, Senior Machine Learning Researcher(DD-AIM,高级机器学习研究员)
Bridging the Question-Answer Gap in Retrieval-Augmented Generation: Hypothetical Prompt Embeddings
University of Primorska(普里莫尔斯卡大学);InnoRenew CoE;Research Centre of the Slovenian Academy of Sciences and Arts(斯洛文尼亚科学与艺术研究院研究中心);The Fran Ramovš Institute(弗兰·拉莫夫什研究所)
Self-Aware Vector Embeddings for Retrieval-Augmented Generation: A Neuroscience-Inspired Framework for Temporal, Confidence-Weighted, and Relational Knowledge
Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generation
Zhongguancun Laboratory(中关村实验室);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);Nanyang Technological University(南洋理工大学);Hong Kong University of Science and Technology(香港科技大学);Beijing Academy of Blockchain and Edge Computing(北京区块链与边缘计算研究院);Department of Computer Science, University of Illinois(伊利诺伊大学计算机科学系)
Structure-Augmented Text Representation Learning for Efficient Knowledge Graph Completion
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院);College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);International Center of Future Science, Jilin University(吉林大学未来科学国际中心)
Lighter And Better: Towards Flexible Context Adaptation For Retrieval Augmented Generation
University of Science and Technology of China(中国科学技术大学);Gaoling School of AI, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Beijing University of Posts and Telecommunications(北京邮电大学)
LaB-RAG: Label Boosted Retrieval Augmented Generation for Radiology Report Generation
Department of Computer Science(计算机科学系);Center for Translational Data Science(转化数据科学中心);Medical Scientist Training Program(医学科学家培训计划);Section of Biomedical Data Science, Department of Medicine(医学部生物医学数据科学部门)