Parametric language models (LMs), which are trained on vast amounts of web data, exhibit remarkable flexibility and capability. However, they still face practical challenges such as hallucinations, difficulty in adapting to new data distributions, and a lack of verifiability. In this position paper, we advocate for retrieval-augmented LMs to replace parametric LMs as the next generation of LMs. By incorporating large-scale datastores during inference, retrieval-augmented LMs can be more reliable, adaptable, and attributable. Despite their potential, retrieval-augmented LMs have yet to be widely adopted due to several obstacles: specifically, current retrieval-augmented LMs struggle to leverage helpful text beyond knowledge-intensive tasks such as question answering, have limited interaction between retrieval and LM components, and lack the infrastructure for scaling. To address these, we propose a roadmap for developing general-purpose retrieval-augmented LMs. This involves a reconsideration of datastores and retrievers, the exploration of pipelines with improved retriever-LM interaction, and significant investment in infrastructure for efficient training and inference.
Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer
Carnegie Mellon University(卡内基梅隆大学);Bosch Research North America(博世北美研究院);Amazon(亚马逊)
知识密集型任务的系统,如开放领域的问答(QA)通常包括两个阶段:从一个大的语料库中有效检索相关的文件和详细阅读所选择的文件,以产生答案。检索器和读取器通常是单独建模的,这需要繁琐的实现,并且很难以端到端的方式进行训练和调整。在本文中,我们重新审视了这种设计,并避开了单独的架构和培训,而是使用单个Transformer来执行Retrieval as Attention(ReAtt),并仅基于来自最终QA任务的监督进行端到端培训。我们第一次证明,一个单一的模型训练的端到端可以实现竞争检索和QA性能,匹配或略优于最先进的分别训练检索和读者。此外,端到端自适应显著提高了其在有监督和无监督环境下对域外数据集的性能,使我们的模型成为知识密集型任务的简单且适应性强的解决方案。代码和模型可在https://github.com/jzbjyb/ReAtt上获得。
VeritasFi: An Adaptable, Multi-tiered RAG Framework for Multi-modal Financial Question Answering
University of Toronto(多伦多大学);McMaster University(麦马斯特大学);McGill University(麦吉尔大学);University of Manitoba(曼尼托巴大学);University of California, Los Angeles(加州大学洛杉矶分校);University of Montreal(蒙特利尔大学);Mila;CUHK(香港中文大学);HKUST(GZ)(香港理工大学(广州));Nanyang Technological University(南洋理工大学);Stanford University(斯坦福大学);CG Matrix Technology Limited(CG矩阵科技有限公司)
Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps
School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院);Central South University(中南大学);Big data and artificial intelligent institute, China Telecom Research Institute(中国电信大数据与人工智能研究院)
DRAK: Unlocking Molecular Insights with Domain-Specific Retrieval-Augmented Knowledge in LLMs
Xiongan Institute of Innovation, Chinese Academy of Sciences(中国科学院雄安创新研究院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Science(中国科学院大学)
Investigating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queries
Yale University(耶鲁大学);University of Pennsylvania(宾夕法尼亚大学);Human Language Technology Center of Excellence(语言技术中心卓越中心);Johns Hopkins University(约翰霍普金斯大学)
CoCR-RAG: Enhancing Retrieval-Augmented Generation in Web Q&A via Concept-oriented Context Reconstruction
University of Southern Queensland(昆士兰大学);University of Technology Sydney(新南威尔士大学);The Hong Kong Polytechnic University(香港理工大学);National University of Singapore(新加坡国立大学);Qatar Computing Research Institute(卡塔尔计算研究所);The Education University of Hong Kong(香港教育大学)
PAKTON: A Multi-Agent Framework for Question Answering in Long Legal Agreements
School of Electrical and Computer Engineering, AILS Laboratory, National Technical University of Athens(电气与计算机工程学院,AILS实验室,国家技术大学雅典);Instituto de Telecomunicações(电信研究所)
MedPromptExtract (Medical Data Extraction Tool): Anonymization and Hi-fidelity Automated data extraction using NLP and prompt engineering
Indian Institute of Technology, Bombay(印度理工学院孟买分校);Sardar Patel Institute of Technology(萨达尔·帕特尔技术学院);Kokilaben Dhirubhai Ambani Hospital and Medical Research Institute(科基拉本·迪鲁巴伊·安巴尼医院与医学研究所)
提出 MedPromptExtract,结合 EIGEN 匿名化、NLP 与基于 Gemini Pro 的 LLM 提示工程,从 AKI 出院小结中高保真提取十二个临床特征。
DataGen: Unified Synthetic Dataset Generation via Large Language Models
University of Notre Dame(诺丁汉大学);Huazhong University of Science and Technology(华中科技大学);MBZUAI;University of Washington(华盛顿大学);Peking University(北京大学);University of Maryland, College Park(马里兰大学学院市分校);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Microsoft Research(微软研究院);Lehigh University(莱斯大学)