In this paper, we establish a benchmark named HalluQA (Chinese Hallucination Question-Answering) to measure the hallucination phenomenon in Chinese large language models. HalluQA contains 450 meticulously designed adversarial questions, spanning multiple domains, and takes into account Chinese historical culture, customs, and social phenomena. During the construction of HalluQA, we consider two types of hallucinations: imitative falsehoods and factual errors, and we construct adversarial samples based on GLM-130B and ChatGPT. For evaluation, we design an automated evaluation method using GPT-4 to judge whether a model output is hallucinated. We conduct extensive experiments on 24 large language models, including ERNIE-Bot, Baichuan2, ChatGLM, Qwen, SparkDesk and etc. Out of the 24 models, 18 achieved non-hallucination rates lower than 50%. This indicates that HalluQA is highly challenging. We analyze the primary types of hallucinations in different types of models and their causes. Additionally, we discuss which types of hallucinations should be prioritized for different types of models.
UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation
Institute for Advanced Algorithms Research（高级算法研究院）；State Key Laboratory of Media Convergence Production Technology and Systems（媒体融合生产技术与系统国家重点实验室）；School of Information, Renmin University of China（中国人民大学信息学院）
Evaluating Chinese Ambiguity Understanding in Large Language Models
Graduate School of Information Science and Technology, The University of Tokyo（东京大学信息科学与技术研究生院）；School of Software Engineering, South China University of Technology（华南理工大学软件学院）
CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patterns
Nanyang Technological University（南洋理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Huazhong University of Science and Technology（华中科技大学）
CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models
The Chinese University of Hong Kong（香港中文大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；LightSpeed Studios, Tencent Inc.（腾讯公司光速工作室）；Sun Yat-Sen University（中山大学）
CHBench: A Chinese Dataset for Evaluating Health in Large Language Models
School of Artificial intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University（吉林大学知识驱动人机智能工程研究中心）；International Center of Future Science, Jilin University（吉林大学未来科学国际中心）
Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetry
College of Computer Science, Inner Mongolia University, China（内蒙古大学计算机学院，中国）；National & Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, China（蒙古语智能信息处理技术国家与地方联合工程研究中心，中国）；NLP²CT Lab, Department of Computer and Information Science, University of Macau, China（澳门大学计算机与信息科学学院NLP²CT实验室，中国）
CPsyExam: A Chinese Benchmark for Evaluating Psychology using Examinations
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Jilin University（吉林大学）；University of Science and Technology of China（中国科学技术大学）；Shenzhen University of Advanced Technology（深圳理工大学）；University of Macau（澳门大学）；Huazhong University of Science and Technology（华中科技大学）
The System Hallucination Scale (SHS): A Human-Centered Instrument for Assessing Perceived Reliability of Large Language Models
Machine Learning and Information Science Group, Medical University of Graz（格拉茨医科大学机器学习与信息科学组）；Human Machine Mind Cooperation, Graz, Austria（格拉茨人类机智合作中心）；MIDATA Cooperative, Zurich, Switzerland（苏黎世MIDATA合作组织）；Human-Centered AI Lab, BOKU University Vienna（维也纳BOKU大学人本AI实验室）
Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratch
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）；University of Chinese Academy of Sciences（中国科学院大学）
EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Models
Lappeenranta-Lahti University of Technology LUT（拉佩兰塔-拉赫蒂技术大学）；University of Oulu（奥卢大学）；Southeast University（东南大学）；Shanghai Jiao Tong University（上海交通大学）；Brno University of Technology（布拉格技术大学）
Fùxì: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generation
School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Lab of Artificial Intelligence for Education, East China Normal University（华东师范大学教育人工智能实验室）；Shanghai Institute of Artificial Intelligence for Education, East China Normal University（华东师范大学上海教育人工智能研究院）
HalluScore: Large Language Model Hallucination Question Answering Benchmark
Department of Information and Computer Science, King Fahd University of Petroleum and Minerals（国王法赫德石油与矿物大学信息与计算机科学系）；SDAIA-KFUPM Joint Research Center for Artificial Intelligence（SDAIA-KFUPM人工智能联合研究中心）
Think More, Hallucinate Less: Mitigating Hallucinations via Dual Process of Fast and Slow Thinking
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Department of Computer Science, National University of Singapore（新加坡国立大学计算机科学系）
FFE-Hallu:Hallucinations in Fixed Figurative Expressions:Benchmark of Idioms and Proverbs in the Persian Language
Tehran Institute for Advanced Studies, Khatam University, Iran（德黑兰高级研究 institute，Khatam 大学，伊朗）；School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran（电气与计算机工程学院，工程学院，德黑兰大学，德黑兰，伊朗）
Small Agent Can Also Rock! Empowering Small Language Models as Hallucination Detector
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；DIRO, Université de Montréal（蒙特利尔大学 DIRO 学院）；Kuaishou（快手）；School of Information, Renmin University of China（中国人民大学信息学院）
Exploring the Generalizability of Factual Hallucination Mitigation via Enhancing Precise Knowledge Utilization
Dept. of Comp. Sci. and Tech.（计算机科学与技术系）；Institute for AI（人工智能研究院）；Tsinghua-Bosch Joint ML Center（清华大学-博世联合机器学习中心）；THBI Lab（THBI实验室）；BNRist Center（BNRist中心）；Tsinghua University（清华大学）
