JuICE：评估LLM裁判识别文化错误的基准
JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors
AI总结：
提出JuICE基准，包含7470个文化语言错误标注的多语言数据集，用于评估LLM裁判在长文本中识别深层文化错误的能力，发现最强模型F1仅0.52且常遗漏本地人易识别的错误。
摘要：
随着大型语言模型（LLM）越来越多地部署给全球用户，它们被整合到不同文化背景下的日常任务中，从起草个人通信到头脑风暴创意想法。这些任务本质上是文化性的：它们需要语境适当性、象征共鸣以及母语者本能依赖的隐性文化期望，这意味着一个回答可能在事实上合理，但对本地读者来说却明显错误。现有的文化基准通过事实验证或规范蕴含方法将文化视为一组扁平的事实，并采用LLM作为裁判，而未检查它们是否能捕捉到这种深层的文化错误。为填补这一空白，我们提出了JuICE（LLM裁判识别文化错误基准），这是一个多语言数据集，包含7470个跨度级别的文化语言错误标注，涵盖来自四个国家（美国、韩国、印度尼西亚和孟加拉国）的1050个查询-响应对，使用英语和这些国家的主要语言。利用JuICE，我们发现即使是最强的LLM裁判在错误跨度检测任务中也仅达到0.52的F1分数。此外，LLM裁判始终会遗漏本地居民容易识别的深层文化错误。我们的研究结果表明，稳健的文化评估必须超越表面级别的检测，转向考虑文化意义的深度和情境性的框架。
英文摘要：
As large language models (LLMs) are increasingly deployed to users around the world, they are integrated into everyday tasks across diverse cultural contexts, from drafting personal communications to brainstorming creative ideas. These tasks are inherently cultural: they require contextual appropriateness, symbolic resonance, and tacit cultural expectations that native speakers draw on instinctively, meaning that a response can be factually plausible yet unmistakably wrong to a local reader. Existing cultural benchmarks have treated culture as a flat set of facts via fact verification or norm entailment methods, and have adopted LLM-as-a-Judge without examining whether they can capture such thick cultural errors. To address this gap, we present JuICE (Benchmark for LLM-Judge in Identifying Cultural Errors), a multilingual dataset of 7,470 span-level annotations of cultural and linguistic errors in long-form LLM responses. It covers 1,050 query-response pairs from four countries (the United States, South Korea, Indonesia, and Bangladesh), in both English and their countries' main languages. Using JuICE, we find that even the strongest LLM-judge achieves only an F1 of 0.52 in the erroneous span detection task. Furthermore, LLM-judges consistently miss thick cultural errors that local residents readily identify. Our findings suggest that robust cultural evaluation must move beyond surface-level detection toward frameworks that account for the depth and situatedness of cultural meaning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
