源自《谁想成为百万富翁？》视频的文化丰富的罗马尼亚语NLP数据集
A Culturally-Rich Romanian NLP Dataset from "Who Wants to Be a Millionaire?" Videos
AI总结：
本文构建了一个源自罗马尼亚版《谁想成为百万富翁？》视频的文化丰富问答数据集，并通过基准、机器翻译和跨语言实验揭示LLM在本土文化问题上的性能短板。
摘要：
大语言模型（LLM）在不同语言和文化语境中表现出参差不齐的性能。本研究介绍了一个新颖的、文化丰富的多语言数据集，该数据集源自罗马尼亚游戏节目《谁想成为百万富翁？》（Vrei să fii Milionar?）的视频录像。我们采用了一种结合光学字符识别（OCR）、自动文本提取和人工核验的创新流程来收集问答对，并使用元数据对其加以丰富，包括问题领域（如生物学、历史）、文化相关性（罗马尼亚特有与国际性）以及难度。在该数据集上对包括罗马尼亚语适配模型在内的最先进LLM进行基准测试，揭示了显著的性能差异：各模型在国际性问题上始终取得更高准确率（80–95%），而在罗马尼亚特有文化问题上的准确率为50–75%。我们进一步通过实验研究这些差异，实验包括将罗马尼亚语问题机器翻译为英语，以及使用可比的法语数据集进行跨语言测试。我们的发现强调了文化语境和数据来源对LLM性能的影响，并为构建稳健、具备文化意识的多语言NLP系统（尤其是在教育领域）提供了实践洞见。该数据集已在Hugging Face上公开发布。
英文摘要：
Large Language Models (LLMs) demonstrate varying performance across languages and cultural contexts. This study introduces a novel, culturally-rich, multilingual dataset derived from video recordings of the Romanian game show "Who Wants to Be a Millionaire?" (Vrei să fii Milionar?). We employed an innovative process combining optical character recognition (OCR), automated text extraction, and manual verification to collect question-answer pairs, enriching them with metadata including question domain (e.g., biology, history), cultural relevance (Romanian-specific vs. international), and difficulty. Benchmarking state-of-the-art LLMs, including Romanian-adapted models, on this dataset revealed significant performance disparities: models consistently achieve higher accuracy (80-95%) on international questions compared to Romanian-specific cultural questions (50-75%). We further investigate these differences through experiments involving machine translation of Romanian questions into English and cross-lingual tests using a comparable dataset in French. Our findings underscore the impact of cultural context and data source on LLM performance and offer practical insights for building robust, culturally-aware multilingual NLP systems, especially in educational domains. The dataset is publicly available at Hugging Face.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
