SDUs DAISY：文化传承的基准测试
SDUs DAISY: A Benchmark for Danish Culture
AI总结：
SDUs DAISY通过文化传承构建新基准，涵盖从古代考古到现代设计的丹麦文化主题，利用语言模型生成问题并人工审核，形成741对封闭式问题-答案对。
摘要：
我们通过文化传承引入了一个新的丹麦文化基准测试Daisy，基于2006年丹麦文化经典中的精选主题。对于文化经典中的每个文物，我们查询相应的维基百科页面，并让语言模型生成随机问题。这在每部作品中产生了一种采样策略，每部作品中包含核心和边缘问题的混合，不仅涵盖主流信息的知识，还深入探讨定义丹麦文化传承的核心基石，由经典委员会定义。每个问题-答案对在最终数据集中经过人工审核或修正，包含741个封闭式问题-答案对，涵盖从公元前1300年的考古发现、18世纪的诗歌和音乐作品到当代流行音乐和丹麦设计与建筑等主题。
英文摘要：
We introduce Daisy, a factual knowledge benchmark for Danish cultural heritage, based on curated topics from the Danish Culture Canon 2006. For each artifact in the culture canon, we query the corresponding Wikipedia page and have a language model generate a diverse set of questions. Within each artifact, we sample both central and peripheral questions, testing not only mainstream information but also the deeper, defining elements of Danish cultural heritage as identified by the Canon committee. Each question-answer pair is manually approved or corrected, yielding a final dataset of 741 closed-ended question-answer pairs ranging from archaeological findings dated to 1300 BCE and 18th-century poems and musical pieces through to contemporary pop music, Danish design, and architecture. Baseline results on our benchmark show that contemporary language models (GPT-OSS-120B & 20B, Llama-3.3-70B, Gemma3-27B, and Mistral-3.1-24B) perform surprisingly poorly on this task, despite the tested factual knowledge being available on Wikipedia. The best-performing model, Llama-3.3-70B, attains only 0.17 BLEU and an F1 score of 0.27, demonstrating the benchmark's difficulty. The questions, model benchmarks, and the evaluation tool are publicly available at https://github.com/schneiderkamplab/SDU-Daisy and under schneiderkamplab/SDU-Daisy on Hugging Face Datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
