SommBench：评估语言模型的品酒师专业能力
SommBench: Assessing Sommelier Expertise of Language Models
AI总结：
SommBench是一个多语言基准，用于评估语言模型的品酒师专业能力，包含三个任务：葡萄酒理论问答、葡萄酒特征补全和食物-葡萄酒配对，测试模型在感官判断上的能力。
摘要：
随着大语言模型的快速发展，系统评估其多语言和多文化能力变得越来越重要。以往的文化评估基准主要集中在可以以语言形式编码的基本文化知识上。在这里，我们提出了SommBench，一个多语言基准，用于评估品酒师专业能力，这是一个深植于嗅觉和味觉感知的领域。虽然语言模型只能通过文本描述学习感官属性，SommBench测试这种文本基础是否足以模拟专家级的感官判断。SommBench包含三个主要任务：葡萄酒理论问答（WTQA）、葡萄酒特征补全（WFC）和食物-葡萄酒配对（FWP）。SommBench支持多种语言：英语、斯洛伐克语、瑞典语、芬兰语、德语、丹麦语、意大利语和西班牙语。这有助于将语言模型的葡萄酒专业知识与其语言能力区分开来。基准数据集是在与专业品酒师和相应语言的母语者密切合作下开发的，最终产生了1,024个葡萄酒理论问答问题、1,000个葡萄酒特征补全示例和1,000个食物-葡萄酒配对示例。我们为最流行的模型提供了结果，包括封闭权重模型如Gemini 2.5，以及开放权重模型，如GPT-OSS和Qwen 3。我们的结果表明，最强大的模型在葡萄酒理论问答上表现良好（封闭权重模型正确率高达97%），但特征补全（峰值为65%）和食物-葡萄酒配对（MCC范围在0到0.39之间）则更具挑战性。这些结果使SommBench成为评估语言模型品酒师专业能力的有趣且具有挑战性的基准。该基准可在https://github.com/sommify/sommbench上公开获取。
英文摘要：
With the rapid advances of large language models, it becomes increasingly important to systematically evaluate their multilingual and multicultural capabilities. Previous cultural evaluation benchmarks focus mainly on basic cultural knowledge that can be encoded in linguistic form. Here, we propose SommBench, a multilingual benchmark to assess sommelier expertise, a domain deeply grounded in the senses of smell and taste. While language models learn about sensory properties exclusively through textual descriptions, SommBench tests whether this textual grounding is sufficient to emulate expert-level sensory judgment. SommBench comprises three main tasks: Wine Theory Question Answering (WTQA), Wine Feature Completion (WFC), and Food-Wine Pairing (FWP). SommBench is available in multiple languages: English, Slovak, Swedish, Finnish, German, Danish, Italian, and Spanish. This helps separate a language model's wine expertise from its language skills. The benchmark datasets were developed in close collaboration with a professional sommelier and native speakers of the respective languages, resulting in 1,024 wine theory question-answering questions, 1,000 wine feature-completion examples, and 1,000 food-wine pairing examples. We provide results for the most popular language models, including closed-weights models such as Gemini 2.5, and open-weights models, such as GPT-OSS and Qwen 3. Our results show that the most capable models perform well on wine theory question answering (up to 97% correct with a closed-weights model), yet feature completion (peaking at 65%) and food-wine pairing show (MCC ranging between 0 and 0.39) turn out to be more challenging. These results position SommBench as an interesting and challenging benchmark for evaluating the sommelier expertise of language models. The benchmark is publicly available at https://github.com/sommify/sommbench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
