HypoTermQA：用于基准测试大语言模型幻觉倾向的假设性术语数据集
HypoTermQA: Hypothetical Terms Dataset for Benchmarking Hallucination Tendency of LLMs
AI总结：
本文提出HypoTermQA框架与数据集，通过LLM生成假设性现象任务并以智能体方式进行幻觉检测，实现领域无关的自动化可扩展基准测试，用于评估和改进LLMs的幻觉倾向。
摘要：
幻觉对大语言模型（LLMs）的可靠性和对齐构成了重大挑战，限制了它们在聊天机器人应用之外被广泛接受。尽管持续努力，幻觉仍然是LLMs中普遍存在的挑战。幻觉的检测本身也是一项艰巨的任务，通常需要人工标注或受限的评估。本文介绍了一个自动化可扩展框架，将基准测试LLMs的幻觉倾向与高效的幻觉检测相结合。我们利用LLMs生成与假设性现象相关的具有挑战性的任务，随后将它们用作智能体进行高效的幻觉检测。该框架与领域无关，允许使用任何语言模型在任何领域进行基准创建或评估。我们引入了公开可用的HypoTermQA基准测试数据集，在该数据集上，最先进模型的性能介于3%到11%之间，评估智能体在幻觉预测中表现出6%的错误率。所提出的框架为测试和改进LLMs提供了机会。此外，它还有潜力生成针对特定领域（如法律、健康和金融）量身定制的基准测试数据集。
英文摘要：
Hallucinations pose a significant challenge to the reliability and alignment of Large Language Models (LLMs), limiting their widespread acceptance beyond chatbot applications. Despite ongoing efforts, hallucinations remain a prevalent challenge in LLMs. The detection of hallucinations itself is also a formidable task, frequently requiring manual labeling or constrained evaluations. This paper introduces an automated scalable framework that combines benchmarking LLMs' hallucination tendencies with efficient hallucination detection. We leverage LLMs to generate challenging tasks related to hypothetical phenomena, subsequently employing them as agents for efficient hallucination detection. The framework is domain-agnostic, allowing the use of any language model for benchmark creation or evaluation in any domain. We introduce the publicly available HypoTermQA Benchmarking Dataset, on which state-of-the-art models' performance ranged between 3% and 11%, and evaluator agents demonstrated a 6% error rate in hallucination prediction. The proposed framework provides opportunities to test and improve LLMs. Additionally, it has the potential to generate benchmarking datasets tailored to specific domains, such as law, health, and finance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
