FELM: Benchmarking Factuality Evaluation of Large Language Models
National University of Singapore(新加坡国立大学);The Hong Kong University of Science and Technology(香港科技大学);Carnegie Mellon University(卡内基梅隆大学);City University of Hong Kong(香港城市大学);Shanghai Jiao Tong University(上海交通大学)
Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, Junxian He
备注:
Accepted by NeurIPS 2023 Track on Datasets and Benchmarks
Assessing factuality of text generated by large language models (LLMs) is an emerging yet crucial research area, aimed at alerting users to potential errors and guiding the development of more reliable LLMs. Nonetheless, the evaluators assessing factuality necessitate suitable evaluation themselves to gauge progress and foster advancements. This direction remains under-explored, resulting in substantial impediments to the progress of factuality evaluators. To mitigate this issue, we introduce a benchmark for Factuality Evaluation of large Language Models, referred to as felm. In this benchmark, we collect responses generated from LLMs and annotate factuality labels in a fine-grained manner. Contrary to previous studies that primarily concentrate on the factuality of world knowledge (e.g.~information from Wikipedia), felm focuses on factuality across diverse domains, spanning from world knowledge to math and reasoning. Our annotation is based on text segments, which can help pinpoint specific factual errors. The factuality annotations are further supplemented by predefined error types and reference links that either support or contradict the statement. In our experiments, we investigate the performance of several LLM-based factuality evaluators on felm, including both vanilla LLMs and those augmented with retrieval mechanisms and chain-of-thought processes. Our findings reveal that while retrieval aids factuality evaluation, current LLMs are far from satisfactory to faithfully detect factual errors.
Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning
College of AI, Tsinghua University(清华大学人工智能学院);Department of Electronic Engineering, BNRist, Tsinghua University(清华大学电子工程系);School of Electronic and Information Engineering, Beijing Jiaotong University(北京交通大学电子信息学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)
Reference Matters: Benchmarking Factual Error Correction for Dialogue Summarization with Fine-grained Evaluation Framework
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Center for Data Science, Peking University(北京大学数据科学中心);The MOE Key Laboratory of Computational Linguistics, Peking University(北京大学计算语言学教育部重点实验室);Huawei Cloud(华为云)
RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Identifying the Achilles' Heel: An Iterative Method for Dynamically Uncovering Factual Errors in Large Language Models
Renmin University of China(中国人民大学);The Chinese University of Hong Kong(香港中文大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学深圳校区);Johns Hopkins University(约翰霍普金斯大学);Nanyang Technological University(南洋理工大学);Xiaohongshu Inc.(小红书公司);Tencent Inc.(腾讯公司)
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZJU-Ant Group Joint Research Center for Knowledge Graphs, Zhejiang University(浙江大学-蚂蚁集团知识图谱联合研究中心);School of Software Technology, Zhejiang University(浙江大学软件学院);Alibaba Group(阿里巴巴集团)