What Should I Cite? A RAG Benchmark for Academic Citation Prediction
Tsinghua University(清华大学);University of Science and Technology of China(中国科学技术大学);Beijing Institute of Technology(北京理工大学);Peking University(北京大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
With the rapid growth of Web-based academic publications, more and more papers are being published annually, making it increasingly difficult to find relevant prior work. Citation prediction aims to automatically suggest appropriate references, helping scholars navigate the expanding scientific literature. Here we present \textbf{CiteRAG}, the first comprehensive retrieval-augmented generation (RAG)-integrated benchmark for evaluating large language models on academic citation prediction, featuring a multi-level retrieval strategy, specialized retrievers, and generators. Our benchmark makes four core contributions: (1) We establish two instances of the citation prediction task with different granularity. Task 1 focuses on coarse-grained list-specific citation prediction, while Task 2 targets fine-grained position-specific citation prediction. To enhance these two tasks, we build a dataset containing 7,267 instances for Task 1 and 8,541 instances for Task 2, enabling comprehensive evaluation of both retrieval and generation. (2) We construct a three-level large-scale corpus with 554k papers spanning many major subfields, using an incremental pipeline. (3) We propose a multi-level hybrid RAG approach for citation prediction, fine-tuning embedding models with contrastive learning to capture complex citation relationships, paired with specialized generation models. (4) We conduct extensive experiments across state-of-the-art language models, including closed-source APIs, open-source models, and our fine-tuned generators, demonstrating the effectiveness of our framework. Our open-source toolkit enables reproducible evaluation and focuses on academic literature, providing the first comprehensive evaluation framework for citation prediction and serving as a methodological template for other scientific domains. Our source code and data are released at https://github.com/LQgdwind/CiteRAG.
VeriCite: Towards Reliable Citations in Retrieval-Augmented Generation via Rigorous Verification
State Key Laboratory of AI Safety, ICT, CAS University of Chinese Academy of Sciences, CAS(人工智能安全国家重点实验室,中国科学院计算技术研究所,中国科学院大学);Meituan Inc.(美团);Baidu Inc.(百度)
Department of Mathematics and Computer Science, University of Catania, Italy(卡塔尼亚大学数学与计算机科学系,意大利);Institute for Cognitive Science and Technology, National Research Council, Italy(意大利国家研究委员会认知科学与技术研究所)
Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study
Department of Data Science & AI, Monash University(数据科学与人工智能系,莫纳什大学);Faculty of Law, Monash University(法学院,莫纳什大学)
Incorporating Q&A Nuggets into Retrieval-Augmented Generation
University of New Hampshire(新罕布什尔大学);University of Pennsylvania(宾夕法尼亚大学);Yale University(耶鲁大学);University of Amsterdam(阿姆斯特丹大学);Human Language Technology Center of Excellence, Johns Hopkins University(约翰霍普金斯大学人类语言技术卓越中心)