When large language models are aligned via supervised fine-tuning, they may encounter new factual information that was not acquired through pre-training. It is often conjectured that this can teach the model the behavior of hallucinating factually incorrect responses, as the model is trained to generate facts that are not grounded in its pre-existing knowledge. In this work, we study the impact of such exposure to new knowledge on the capability of the fine-tuned model to utilize its pre-existing knowledge. To this end, we design a controlled setup, focused on closed-book QA, where we vary the proportion of the fine-tuning examples that introduce new knowledge. We demonstrate that large language models struggle to acquire new factual knowledge through fine-tuning, as fine-tuning examples that introduce new knowledge are learned significantly slower than those consistent with the model's knowledge. However, we also find that as the examples with new knowledge are eventually learned, they linearly increase the model's tendency to hallucinate. Taken together, our results highlight the risk in introducing new factual knowledge through fine-tuning, and support the view that large language models mostly acquire factual knowledge through pre-training, whereas fine-tuning teaches them to use it more efficiently.
AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs
Information and Computer Science Department, King Fahd University of Petroleum and Minerals(信息与计算机科学系,国王法赫德石油和矿物大学);SDAIA-KFUPM Joint Research Center for Artificial Intelligence(SDAIA-KFUPM人工智能联合研究中心)
The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);DIRO, Université de Montréal(蒙特利尔大学计算机科学与运筹学系);School of Information, Renmin University of China(中国人民大学信息资源管理学院)
HalluScore: Large Language Model Hallucination Question Answering Benchmark
Department of Information and Computer Science, King Fahd University of Petroleum and Minerals(国王法赫德石油与矿物大学信息与计算机科学系);SDAIA-KFUPM Joint Research Center for Artificial Intelligence(SDAIA-KFUPM人工智能联合研究中心)
Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMs
University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);Electronic Science Research Institute of China Electronics(中国电子科技集团电子科学研究院)
Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting
Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);State Key Laboratory of Computer Science(计算机科学国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学)
Can LLM be a Good Path Planner based on Prompt Engineering? Mitigating the Hallucination for Path Planning
College of Computer Science(计算机科学学院);Sichuan Normal University(四川师范大学);School of Information and Software Engineering(信息与软件工程学院);University of Electronic Science and Technology of China(电子科技大学)
Bridging External and Parametric Knowledge: Mitigating Hallucination of LLMs with Shared-Private Semantic Synergy in Dual-Stream Knowledge
Beijing Institute of Technology, China(北京理工大学);Beijing University of Posts and Telecommunications, China(北京邮电大学);Meituan, China(美团);The Open University, UK(开放大学)
Why Fine-Tuning Encourages Hallucinations and How to Fix It
Hebrew University of Jerusalem(海法大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Technion – Israel Institute of Technology(技术学院 – 以色列理工学院);University of Southern California(南加州大学)
Towards Detecting LLMs Hallucination via Markov Chain-based Multi-agent Debate Framework
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Gaoling School of Artifical Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院)
AutoHall: Automated Factuality Hallucination Dataset Generation for Large Language Models
AGI Institute, School of Computer Science, Shanghai Jiao Tong University(AGI研究院,计算机科学学院,上海交通大学);Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3(上海可信数据流通与治理Web3重点实验室);School of Media & Communication, Shanghai Jiao Tong University(媒体与传播学院,上海交通大学)
TraceDet: Hallucination Detection from the Decoding Trace of Diffusion Large Language Models
Department of Engineering Science, University of Oxford, UK(牛津大学工程科学系);Hasso Plattner Institute, University of Potsdam(波茨坦大学哈索普兰特纳研究所);Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(姆罕默德·本·拉希德智能大学)