The increased use of large language models (LLMs) across a variety of real-world applications calls for automatic tools to check the factual accuracy of their outputs, as LLMs often hallucinate. This is difficult as it requires assessing the factuality of free-form open-domain responses. While there has been a lot of research on this topic, different papers use different evaluation benchmarks and measures, which makes them hard to compare and hampers future progress. To mitigate these issues, we developed OpenFactCheck, a unified framework, with three modules: (i) RESPONSEEVAL, which allows users to easily customize an automatic fact-checking system and to assess the factuality of all claims in an input document using that system, (ii) LLMEVAL, which assesses the overall factuality of an LLM, and (iii) CHECKEREVAL, a module to evaluate automatic fact-checking systems. OpenFactCheck is open-sourced (https://github.com/mbzuai-nlp/openfactcheck) and publicly released as a Python library (https://pypi.org/project/openfactcheck/) and also as a web service (http://app.openfactcheck.com). A video describing the system is available at https://youtu.be/-i9VKL0HleI.
Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulness
Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);University of California, Merced(加州大学默塞德分校);University of Science and Technology of China(中国科学技术大学)
MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs
Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100085, China(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, China(中国科学院大学网络安全学院)
HGOT: Hierarchical Graph of Thoughts for Retrieval-Augmented In-Context Learning in Factuality Evaluation
Smith School of Business, Queen’s University(女王大学史密斯商学院);Ingenuity Labs Research Institute, Queen’s University(女王大学 Ingenuity 实验室研究所);Department of Electrical and Computer Engineering, Queen’s University(女王大学电气与计算机工程系)
Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation
National Biomarker Centre, CRUK-MI, Univ. of Manchester(英国曼彻斯特大学CRUK-MI国家生物标志物中心);Idiap Research Institute(Idiap研究所);Inflamalps SA(因弗马尔普斯公司);Department of Computer Science, Univ. of Manchester(英国曼彻斯特大学计算机科学系)
Beyond Under-Alignment: Atomic Preference Enhanced Factuality Tuning for Large Language Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
OpenFActScore: Open-Source Atomic Evaluation of Factuality in Text Generation
Philipps-Universität Marburg(马尔堡大学);German Research Centre for Artificial Intelligence (DFKI)(德国人工智能研究中心);Centre for European Research in Trusted AI (CERTAIN)(可信AI欧洲研究中心);Saarland University(萨尔兰大学)
Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning
College of AI, Tsinghua University(清华大学人工智能学院);Department of Electronic Engineering, BNRist, Tsinghua University(清华大学电子工程系);School of Electronic and Information Engineering, Beijing Jiaotong University(北京交通大学电子信息学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)
Does Finetuning with Scientific Data Increase Hallucinations? A Multi-domain Factuality Evaluation of LLMs
Deutsches Forschungszentrum für Künstliche Intelligenz GmbH(德国人工智能研究中心 GmbH);Technische Universität Berlin(柏林技术大学);Humboldt-Universität zu Berlin(柏林洪堡大学)
FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMs
UCLA(美国大学洛杉矶分校);CUHK(香港中文大学);HKUST (GZ)(香港科技大学(广州));HKUST(香港科技大学);Tsinghua University(清华大学);ECNU(华东师范大学);NVIDIA(英伟达);UCL(伦敦大学学院);HKU(香港大学);University of Cambridge(剑桥大学)
FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence
The University of Texas at Austin(德克萨斯大学奥斯汀分校);Massachusetts Institute of Technology(麻省理工学院);University of Duisburg-Essen(杜伊斯堡-埃森大学);Institute for AI in Medicine, University Hospital Essen(埃森大学医院医学人工智能研究所);Georgia Institute of Technology(佐治亚理工学院);Northeastern University(东北大学)
Structured Outputs Enable General-Purpose LLMs to be Medical Experts
University of Bristol(布里斯托大学);Lehigh University(利哈伊大学);The University of Queensland(昆士兰大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);University of California, Merced(加利福尼亚大学默塞德分校)
Saarland University(萨尔大学);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);University of Cambridge(剑桥大学);University of Edinburgh(爱丁堡大学);Zhejiang University(浙江大学);Tencent YouTu Lab(腾讯优图实验室)