Hallucinations pose a significant challenge to the reliability of large language models (LLMs) in critical domains. Recent benchmarks designed to assess LLM hallucinations within conventional NLP tasks, such as knowledge-intensive question answering (QA) and summarization, are insufficient for capturing the complexities of user-LLM interactions in dynamic, real-world settings. To address this gap, we introduce HaluEval-Wild, the first benchmark specifically designed to evaluate LLM hallucinations in the wild. We meticulously collect challenging (adversarially filtered by Alpaca) user queries from ShareGPT, an existing real-world user-LLM interaction datasets, to evaluate the hallucination rates of various LLMs. Upon analyzing the collected queries, we categorize them into five distinct types, which enables a fine-grained analysis of the types of hallucinations LLMs exhibit, and synthesize the reference answers with the powerful GPT-4 model and retrieval-augmented generation (RAG). Our benchmark offers a novel approach towards enhancing our comprehension of and improving LLM reliability in scenarios reflective of real-world interactions. Our benchmark is available at https://github.com/HaluEval-Wild/HaluEval-Wild.
Investigating Symbolic Triggers of Hallucination in Gemma Models Across HaluEval and TruthfulQA
Center for Artificial Intelligence in Medicine, Imaging and Forensics, Sharda University, Greater Noida, India(人工智能医学与影像及法医学中心,沙达大学,印度Greater Noida);University of Maryland, Baltimore County, Baltimore, MD, USA(马里兰大学巴尔的摩县)
The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);DIRO, Université de Montréal(蒙特利尔大学计算机科学与运筹学系);School of Information, Renmin University of China(中国人民大学信息资源管理学院)
How Far Can You Get Without a GPU? A Systematic Benchmark of Lightweight Hallucination Detection Across Question Answering, Dialogue, and Summarisation
ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);HKGAI under InnoHK(香港人工智能研究院(InnoHK框架下));Hong Kong University of Science and Technology(香港科技大学)
MALM: A Multi-Information Adapter for Large Language Models to Mitigate Hallucination
Beijing Institute of Technology(北京理工大学);The Open University(开放大学);Beijing Information Science and Technology University(北京信息科技大学);The Hong Kong Polytechnic University(香港理工大学)
Addressing Hallucinations in Language Models with Knowledge Graph Embeddings as an Additional Modality
Institute for Clarity in Documentation(文献清晰研究所);Inria Paris-Rocquencourt(法国国家信息与自动化研究所巴黎-罗康校区);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室)
Hallucinations in Large Multilingual Translation Models
Instituto de Telecomunicações(电信研究所);Instituto Superior Técnico, University of Lisbon(里斯本大学高等技术学院);University of Edinburgh(爱丁堡大学);CentraleSupélec, Université Paris-Saclay(巴黎萨克雷大学中央理工-高等电力学院);Unbabel(Unbabel公司)
Mitigating LLM Hallucinations through Domain-Grounded Tiered Retrieval
Computational Unit, Z.H. College of Engineering & Technology, Aligarh Muslim University, India(计算单元,Z.H. 工程技术学院,阿利加尔穆斯林大学,印度);Interdisciplinary Center for Artificial Intelligence, Aligarh Muslim University, India(跨学科人工智能中心,阿利加尔穆斯林大学,印度);Department of Computer Engineering, Aligarh Muslim University, India(计算机工程系,阿利加尔穆斯林大学,印度)
On Synthesizing Data for Context Attribution in Question Answering
NEC Laboratories Europe(NEC欧洲实验室);NEC Laboratories America(NEC美国实验室);NEC Corporation(NEC公司);University College London(伦敦大学学院);KU Leuven(鲁汶大学);Epimind;Center for Artificial Intelligence and Data Science, University of Würzburg(维尔茨堡大学人工智能与数据科学中心);CAIR, Ss. Cyril and Methodius University of Skopje(斯科普里圣西里尔和美多迪乌斯大学CAIR)
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology(麻省理工学院电子工程与计算机科学系);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室);IBM Research(IBM研究院)