Recent advancements in large language models have demonstrated significant potential in the automated construction of knowledge graphs from unstructured text. This paper builds upon our previous work [16], which evaluated various models using metrics like precision, recall, F1 score, triple matching, and graph matching, and introduces a refined approach to address the critical issues of hallucination and omission. We propose an enhanced evaluation framework incorporating BERTScore for graph similarity, setting a practical threshold of 95% for graph matching. Our experiments focus on the Mistral model, comparing its original and fine-tuned versions in zero-shot and few-shot settings. We further extend our experiments using examples from the KELM-sub training dataset, illustrating that the fine-tuned model significantly improves knowledge graph construction accuracy while reducing the exact hallucination and omission. However, our findings also reveal that the fine-tuned models perform worse in generalization tasks on the KELM-sub dataset. This study underscores the importance of comprehensive evaluation metrics in advancing the state-of-the-art in knowledge graph construction from textual data.
NTS-CoT: Mitigating Hallucinations in LLM-based News Timeline Summarization with Chain-of-Thought Reasoning
Central South University(中南大学);Tsinghua University(清华大学);Nanjing University(南京大学);Suzhou Aerospace Information Research Institute(苏州空天信息研究院);McGill University(麦吉尔大学)
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Tencent AI lab(腾讯AI实验室);Soochow University(苏州大学);Zhejiang University(浙江大学);Renmin University of China(中国人民大学);Nanyang Technological University(南洋理工大学);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所)
SafeLLM: Extraction as a Hallucination-Resistant Alternative to Rewriting in Safety-Critical Settings
Institute of Health Informatics, University College London(伦敦大学学院健康信息学研究所);National Hospital for Neurology and Neurosurgery(国家神经内科与神经外科医院);Somerset NHS Foundation Trust(萨默塞特NHS基金会信托);King's College Hospital(国王学院医院);King's College London(伦敦国王学院)
Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategies
Shanghai Institute of Infectious Disease and Biosecurity, Fudan University(复旦大学附属上海市感染与生物安全研究院);Shanghai Health Development Research Center(Shanghai Medical Information Center)(上海卫生发展研究中心(上海市医学信息中心));Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
PankRAG: Enhancing Graph Retrieval via Globally Aware Query Resolution and Dependency-Aware Reranking Mechanism
College of Computer Science, Beijing university of technology, Beijing, China(北京理工大学计算机科学学院);Department of Computer Science and Engineering, Bologna University, Bologna, Italy(博洛尼亚大学计算机科学与工程系)
Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling
College of Computer, National University of Defense Technology(国防科技大学计算机学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)