VideoNorms：视频语言模型文化意识基准测试
VideoNorms: Benchmarking Cultural Awareness of Video Language Models
AI总结：
本研究推出VideoNorms数据集评估VideoLLMs的文化规范意识，发现模型中文表现逊于英文、提供非语言证据更困难，视频模态必要且规模扩大无分类分数提升，成果助力文化导向的视频模型训练与评估。
摘要：
随着视频大语言模型（VideoLLMs）在全球部署，评估其跨文化语境推理能力至关重要。为推进VideoLLMs的文化规范意识评估，我们推出VideoNorms数据集，该数据集包含来自美国和中国热门电视剧的文化规范标注，标注内容涵盖遵守或违反标签以及非语言证据。通过人机协作框架，每个条目先由大型VideoLLM标注，再由至少三名在目标文化中有丰富生活经验的训练过的单一文化标注员审核，最终得到包含3000多个人类判断的数据集。人类验证显示，美国和中国的规范提取性能存在差异，这提醒人们警惕针对训练数据中代表性不足文化的完全自动化方法。对7个开放权重VideoLLMs性能的分层线性建模分析显示：1）模型在中文上的表现比在美国差，尤其是在规范遵守预测方面；2）在规范遵守/违反预测中，模型提供非语言证据比提供语言证据更困难。消融研究证实，视频模态对准确性能确实必要，且模型规模扩大不会带来分类分数提升。我们的发现和数据有助于基于文化的视频模型训练与评估。
英文摘要：
As Video Large Language Models (VideoLLMs) are deployed globally, it is important to assess their ability to reason across cultural contexts. To advance cultural norm awareness evaluation in VideoLLMs, we introduce VideoNorms, a dataset of cultural norm annotations from popular US and Chinese TV shows annotated with adherence or violation labels and (non-)verbal evidence. Through a human-AI collaboration framework, each item was first annotated by a large VideoLLM, and then reviewed by at least three trained monocultural annotators with significant lived experience in the target culture, resulting in a dataset of over 3,000 human judgments. Human verification showed disparity in US and Chinese norm extraction performance, cautioning against fully automatic approaches cultures under-represented in training data. Hierarchical linear modeling analysis of $7$ open-weight VideoLLMs' performance revealed that: 1) models perform worse in Chinese compared to US, particularly for norm adherence prediction; 2) models have more difficulty in providing non-verbal evidence compared to verbal evidence for norm adherence/violation predictions. Ablation studies confirm video modality is indeed necessary for accurate performance, and scaling model size does not yield classification score improvements. Our findings and data contribute to culturally grounded video model training and evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
