MUCH：一个多语言声称幻觉基准
MUCH: A Multilingual Claim Hallucination Benchmark
AI总结：
MUCH是一个多语言声明幻觉基准，通过提供24个生成logits和高效分割算法，评估声明级不确定性量化方法在现实条件下的性能和效率。
摘要：
声明级不确定性量化（UQ）是一种有前景的方法，用于缓解大型语言模型（LLMs）可靠性不足的问题。我们介绍了MUCH，第一个声明级UQ基准，旨在公平且可重复地评估未来方法在现实条件下的表现。它包含4,873个样本，涵盖四种欧洲语言（英语、法语、西班牙语和德语）以及四种经过指令微调的开源LLM。与以往的声明级基准不同，我们发布了每token 24个生成logits，这有助于开发未来白盒方法而无需重新生成数据。此外，与以往依赖手动或LLM基于分割的基准不同，我们提出了一种新的确定性算法，能够使用仅0.2%的LLM生成时间来分割声明。这使我们的分割方法适用于LLM输出的实时监控，确保MUCH在现实部署约束下评估UQ方法。最后，我们的评估显示，当前方法在性能和效率方面仍有较大的改进空间。
英文摘要：
Claim-level Uncertainty Quantification (UQ) is a promising approach to mitigate the lack of reliability in Large Language Models (LLMs). We introduce MUCH, the first claim-level UQ benchmark designed for fair and reproducible evaluation of future methods under realistic conditions. It includes 4,873 samples across four European languages (English, French, Spanish, and German) and four instruction-tuned open-weight LLMs. Unlike prior claim-level benchmarks, we release 24 generation logits per token, facilitating the development of future white-box methods without re-generating data. Moreover, in contrast to previous benchmarks that rely on manual or LLM-based segmentation, we propose a new deterministic algorithm capable of segmenting claims using as little as 0.2% of the LLM generation time. This makes our segmentation approach suitable for real-time monitoring of LLM outputs, ensuring that MUCH evaluates UQ methods under realistic deployment constraints. Finally, our evaluations show that current methods still have substantial room for improvement in both performance and efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
