Large language models are powerful text processors and reasoners, but are still subject to limitations including outdated knowledge and hallucinations, which necessitates connecting them to the world. Retrieval-augmented large language models have raised extensive attention for grounding model generation on external knowledge. However, retrievers struggle to capture relevance, especially for queries with complex information needs. Recent work has proposed to improve relevance modeling by having large language models actively involved in retrieval, i.e., to improve retrieval with generation. In this paper, we show that strong performance can be achieved by a method we call Iter-RetGen, which synergizes retrieval and generation in an iterative manner. A model output shows what might be needed to finish a task, and thus provides an informative context for retrieving more relevant knowledge which in turn helps generate a better output in the next iteration. Compared with recent work which interleaves retrieval with generation when producing an output, Iter-RetGen processes all retrieved knowledge as a whole and largely preserves the flexibility in generation without structural constraints. We evaluate Iter-RetGen on multi-hop question answering, fact verification, and commonsense reasoning, and show that it can flexibly leverage parametric knowledge and non-parametric knowledge, and is superior to or competitive with state-of-the-art retrieval-augmented baselines while causing fewer overheads of retrieval and generation. We can further improve performance via generation-augmented retrieval adaptation.
Bridging External and Parametric Knowledge: Mitigating Hallucination of LLMs with Shared-Private Semantic Synergy in Dual-Stream Knowledge
Beijing Institute of Technology, China（北京理工大学）；Beijing University of Posts and Telecommunications, China（北京邮电大学）；Meituan, China（美团）；The Open University, UK（开放大学）
RAIR: Retrieval-Augmented Iterative Refinement for Chinese Spelling Correction
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhongke Fanyu Technology Co., Ltd（中科泛语科技有限公司）
Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Key Laboratory of AI Safety, Chinese Academy of Sciences（中国科学院人工智能安全重点实验室）
A + B: A General Generator-Reader Framework for Optimizing LLMs to Unleash Synergy Potential
University of Science and Technology of China（中国科学技术大学）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Singapore Management University（新加坡管理大学）；Beijing Institute of Technology（北京理工大学）；Aarhus University（奥胡斯大学）
Exploring the Human-LLM Synergy in Advancing Theory-driven Qualitative Analysis
Department of Computer Science, National University of Singapore（计算机科学系，国立新加坡大学）；Amazon Research, Amazon.com Inc（亚马逊研究，亚马逊公司）；Department of Social Work, National University of Singapore（社会工作系，国立新加坡大学）
Self-evolving Agents with reflective and memory-augmented abilities
East China Jiao Tong University（东中国交通大学）；University of Minnesota - Twin Cities（明尼苏达大学-双城分校）；Emory University（埃默里大学）；University of Electronic Science and Technology of China（电子科技大学）；Guangdong University of Technology（广东工业大学）；Amazon（亚马逊）；Washington University（华盛顿大学）；Xiamen University（厦门大学）；University of Bristol（布里斯托尔大学）；University of Toronto（多伦多大学）
Hefei Institutes of Physical Science, Chinese Academy of Sciences（合肥物理科学研究院，中国科学院）；University of Science and Technology of China（中国科学技术大学）；Changzhou University（常州大学）
Beyond Entity Alignment: Towards Complete Knowledge Graph Alignment via Entity-Relation Synergy
School of Information Science and Engineering, Yanshan University（燕山大学信息科学与工程学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beihang University（北京航空航天大学）
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Salesforce Research（赛富时研究院）
Existing KBQA approaches, despite achieving strong performance on i.i.d. test data, often struggle in generalizing to questions involving unseen KB schema items. Prior ranking-based approaches have shown some success in generalization, but suffer from the coverage issue. We present RnG-KBQA, a Rank-and-Generate approach for KBQA, which remedies the coverage issue with a generation model while preserving a strong generalization capability. Our approach first uses a contrastive ranker to rank a set of candidate logical forms obtained by searching over the knowledge graph. It then introduces a ta…
Amazon Inc.（亚马逊公司）；Salesforce Research（赛富时研究院）；University of Pittsburgh（匹兹堡大学）；Chinese University of Hong Kong（香港中文大学）
Keyphrase generation is the task of automatically predicting keyphrases given a piece of long text. Despite its recent flourishing, keyphrase generation on non-English languages haven't been vastly investigated. In this paper, we call attention to a new setting named multilingual keyphrase generation and we contribute two new datasets, EcommerceMKP and AcademicMKP, covering six languages. Technically, we propose a retrieval-augmented method for multilingual keyphrase generation to mitigate the data shortage problem in non-English languages. The retrieval-augmented model leverages keyphrase ann…
Cognition on Graph: Navigating Massive Knowledge Space via Cognitive Cycles and Bidirectional Graph-Text Synergy
Beijing University of Posts and Telecommunications（北京邮电大学）；Zhongguancun Academy（中关村学院）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统全国重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Facebook AI Research (FAIR)（脸书人工智能研究院）；University of California, Berkeley（加州大学伯克利分校）
Many visual scenes contain text that carries crucial information, and it is thus essential to understand text in images for downstream reasoning tasks. For example, a deep water label on a warning sign warns people about the danger in the scene. Recent work has explored the TextVQA task that requires reading and understanding text in images to answer a question. However, existing approaches for TextVQA are mostly based on custom pairwise fusion mechanisms between a pair of two modalities and are restricted to a single prediction step by casting TextVQA as a classification task. In this work, w…
Graph Counselor: Adaptive Graph Exploration via Multi-Agent Synergy to Enhance LLM Reasoning
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Mathematics, Harbin Institute of Technology（哈尔滨工业大学数学学院）；Department of Control Science and Engineering, Harbin Institute of Technology（哈尔滨工业大学控制科学与工程学院）
Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questions
University of Virginia（弗吉尼亚大学）；National Library of Medicine（美国国家医学图书馆）；National Institutes of Health（美国国立卫生研究院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
FAIR-RAG: Faithful Adaptive Iterative Refinement for Retrieval-Augmented Generation
Sharif University of Technology（谢里夫理工大学）；Iran University of Science and Technology（伊朗科学技术大学）；Noor Avaran Jelvehaye Maanaei Najm Co., Ltd.（诺尔·阿瓦兰·杰拉维耶·马纳埃尼纳姆公司）
Chain of Evidence: Pixel-Level Visual Attribution for Iterative Retrieval-Augmented Generation
National Engineering Research Center for Software Engineering, Peking University（软件工程国家级工程研究中心，北京大学）；City University of Hong Kong（香港城市大学）；Peking University（北京大学）；Tencent Technology（腾讯科技）
提出Chain of Evidence (CoE)框架，利用视觉语言模型直接对检索到的文档截图进行推理，输出精确边界框以可视化完整推理链，解决迭代检索增强生成中的粗粒度归因和视觉语义丢失问题。
Transformer based language models (LMs) demonstrate increasing performance with scale across a wide variety of tasks. Scale alone however cannot enable models to solve tasks that require access to ephemeral, changing, or private data that was unavailable at training time. Many useful tasks may also benefit from LMs being able to access APIs that read or modify state. In this work, we present Tool Augmented Language Models (TALM), combining a text-only approach to augment language models with non-differentiable tools, and an iterative "self-play" technique to bootstrap performance starting from…
LaB-RAG: Label Boosted Retrieval Augmented Generation for Radiology Report Generation
Department of Computer Science（计算机科学系）；Center for Translational Data Science（转化数据科学中心）；Medical Scientist Training Program（医学科学家培训计划）；Section of Biomedical Data Science, Department of Medicine（医学部生物医学数据科学部门）
Information Sciences Institute（信息科学研究所）；University of Southern California（南加州大学）
We present a weakly-supervised data augmentation approach to improve Named Entity Recognition (NER) in a challenging domain: extracting biomedical entities (e.g., proteins) from the scientific literature. First, we train a neural NER (NNER) model over a small seed of fully-labeled examples. Second, we use a reference set of entity names (e.g., proteins in UniProt) to identify entity mentions with high precision, but low recall, on an unlabeled corpus. Third, we use the NNER model to assign weak labels to the corpus. Finally, we retrain our NNER model iteratively over the augmented training set…
University of Macau（澳门大学）；Xiamen University（厦门大学）；Peking University（北京大学）；Hanyang University（翰阳大学）；University of Liverpool（利物浦大学）；Zhejiang University（浙江大学）
