大型语言模型能否理解隐性文化价值观？——用CQ-Bench基准测试LLMs的文化智能
Can LLMs Grasp Implicit Cultural Values? Benchmarking LLMs' Cultural Intelligence with CQ-Bench
AI总结：
本文提出CQ-Bench基准，通过三个递进任务评估LLMs从对话中推断隐性文化价值观的能力，发现前沿模型在态度检测上仍逊于人类，而小模型微调可显著提升性能。
摘要：
文化智能（CQ）指的是理解陌生文化背景的能力，这是大型语言模型（LLMs）有效服务全球多样化用户的关键技能。现有研究往往聚焦于明确陈述的文化规范，却未能捕捉日常对话中常见的微妙、隐性的价值观。为弥补这一空白，我们引入了CQBench，一个专门设计用于评估LLMs从自然对话情境中推断隐性文化价值观能力的基准。CQBench包含基于世界价值观调查和全球舆情的多角色对话故事，主题涵盖伦理、宗教、社会等。我们的自动数据集构建流程整合了严格的验证程序（包含性、一致性和隐晦性检查），最终验证中实现了94.5%的人机一致性。为利用CQBench数据，我们设计了三个复杂度递增的任务：态度检测、价值选择和价值提取。这些任务评估模型能否检测态度并识别自然对话中嵌入的价值观，而非依赖显性文化知识。我们发现，尽管像o1这样的前沿模型在价值选择（F1为0.809）上达到人类水平，但在细微态度检测（F1为0.622）上仍显不足。值得注意的是，仅用500个文化丰富的样本微调较小的LLaMA-3.2-3B模型，性能提升了超过10%，在某些情况下甚至优于o3-mini。利用CQ-Bench，我们提供了对当前LLMs文化智能研究挑战的见解，并提出了增强LLMs跨文化推理能力的实用途径。
英文摘要：
Cultural Intelligence (CQ) refers to the ability to understand unfamiliar cultural contexts, a crucial skill for large language models (LLMs) to effectively engage with globally diverse users. Existing studies often focus on explicitly stated cultural norms, but fail to capture the subtle, implicit values that are common in daily conversation. To address this gap, we introduce CQBench, a benchmark specifically designed to assess LLMs' capability to infer implicit cultural values from natural conversational contexts. CQBench consists of multi character conversation based stories using values from the World Value Survey and the GlobalOpinions, with topics including ethical, religious, social, etc. Our automatic dataset construction pipeline integrates rigorous validation procedures (incorporation, consistency, and implicitness checks), achieving a 94.5% human model agreement in the final validation. To leverage CQBench data, we design three tasks of increasing complexity: attitude detection, value selection, and value extraction. These tasks evaluate whether models can detect attitude and recognize values embedded within natural dialogues rather than relying on explicit cultural knowledge. We find that while frontier models like o1 reach human level performance in value selection (0.809 F1), they still fall short in nuanced attitude detection (0.622 F1). Notably, finetuning a smaller LLaMA-3.2-3B on only 500 culturally rich examples improves performance by over 10%, even outperforming o3-mini in some cases. Using CQ-Bench, we provide insights into the current challenges in LLMs' CQ research and suggest practical pathways for enhancing LLMs' cross-cultural reasoning abilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
