RiDiC编年史：生成具有受控流行度分布的数据集用于长文本事实性评估
The Chronicles of RiDiC: Generating Datasets with Controlled Popularity Distribution for Long-form Factuality Evaluation
AI总结：
本文提出了一种生成多语言实体数据集的可配置流程，用于评估LLM长文本生成的事实性，通过RiDiC数据集展示，包含3000个实体，涵盖不同流行度层级。
摘要：
我们提出了一种可配置的流程，利用维基百科和维基数据生成具有指定特征（如领域、地理位置和流行度）的多语言实体数据集。这些数据集旨在评估LLM长文本生成的事实性，从而补充基于短文本问答数据集的评估。我们展示了RiDiC数据集作为该方法的示例。RiDiC包含来自三个领域的3000个实体——河流、自然灾害和汽车型号——跨越不同的流行度层级。每个实体都配有其地理位置、英文和中文名称（如可用）以及相关的英文和中文维基百科内容，用于评估LLM的响应。关于RiDiC实体的生成来自三种LLM的英文和中文生成。这些生成随后通过第三方事实性检查器进行评估，结果显示我们的数据集中的实体甚至让前沿模型产生了幻觉。为促进LLM多语言长文本事实性评估，代码、数据和生成/评估脚本已发布。
英文摘要：
We present a configurable pipeline for generating multilingual sets of entities with specified characteristics, such as domain, geographical location and popularity, using data from Wikipedia and Wikidata. These datasets are intended for evaluating the factuality of LLMs' long-form generation, thereby complementing evaluation based on short-form QA datasets. We present the RiDiC dataset as an example of this approach. RiDiC contains 3,000 entities from three domains -- rivers, natural disasters, and car models -- spanning different popularity tiers. Each entity is accompanied by its geographical location, English and Chinese names (if available) and relevant English and Chinese Wikipedia content, which is used to evaluate LLMs' responses. Generations about RiDiC entities were obtained from three LLMs in English and Chinese. These were then evaluated using a third-party factuality checker, which showed that entities from our dataset caused even frontier models to hallucinate. To facilitate the evaluation of LLMs' long-form factuality in multiple languages, the code, data, and generation/evaluation scripts have been released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
