arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.08608cs.CLcs.AI

MMA-ASIA:面向文化基础评估的多语言多模态对齐框架

MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation

Singapore University of Technology and Design(新加坡科技设计大学);Agency for Science, Technology and Research, Singapore(新加坡科技研究局);Indian Institute of Technology Delhi(印度理工学院德里分校);Alibaba DAMO Academy(阿里巴巴达摩院);Microsoft Research Asia(微软亚洲研究院);Shanghai University of Finance and Economics(上海财经大学);Inner Mongolia University(内蒙古大学);Kyoto University(京都大学);Jiangxi Normal University(江西师范大学);Korea University(韩国大学);Nanyang Technological University(南洋理工大学)
Weihua Zheng, Zhengyuan Liu, Tanmoy Chakraborty, Weiwen Xu, Xiaoxue Gao, Bryan Chen Zhengyu Tan, Bowei Zou, Chang Liu, Yujia Hu, Xing Xie, Xiaoyuan Yi, Jing Yao… 展开作者

Weihua Zheng, Zhengyuan Liu, Tanmoy Chakraborty, Weiwen Xu, Xiaoxue Gao, Bryan Chen Zhengyu Tan, Bowei Zou, Chang Liu, Yujia Hu, Xing Xie, Xiaoyuan Yi, Jing Yao, Chaojun Wang, Long Li, Rui Liu, Huiyao Liu, Koji Inoue, Ryuichi Sumida, Tatsuya Kawahara, Fan Xu, Lingyu Ye, Wei Tian, Dongjun Kim, Jimin Jung, Jaehyung Seo, Nadya Yuki Wangsajaya, Pham Minh Duc, Ojasva Saxena, Palash Nandi, Xiyan Tao, Wiwik Karlina, Tuan Luong, Keertana Arun Vasan, Roy Ka-Wei Lee, Nancy F. Chen

AI总结:

提出MMA-ASIA框架,含27,000题多语言多模态基准和五维评估协议,检测文化意识与捷径学习,并分析跨语言模态差异原因。

摘要:

大型语言模型(LLMs)现已全球使用,但其多模态理解和推理能力在西方、高资源环境之外往往下降。我们提出MMA-ASIA,一个全面评估LLMs文化意识、聚焦亚洲语境的框架。MMA-ASIA以一个人工策划、多语言、多模态对齐的选择题基准为核心,覆盖8个亚洲国家和10种语言,包含27,000个问题;其中超过79%的问题需要基于文化语境的多步推理,超越了简单记忆。据我们所知,这是首个在三种模态(文本、图像(视觉问答)和语音)输入级别对齐的数据集,从而能够直接测试跨模态迁移。基于该基准,我们提出一个五维评估协议,衡量:(i)跨国家的文化意识差异,(ii)跨语言一致性,(iii)跨模态一致性,(iv)文化知识泛化,以及(v)基础有效性。为确保严格评估,文化意识基础验证模块通过检查所需文化知识是否支持正确答案来检测“捷径学习”。最后,通过比较模型分析、注意力追踪和创新的视觉消融前缀重放(VPR)方法,我们探究模型为何在不同语言和模态间产生分歧,为构建文化可靠的多模态LLMs提供可操作见解。

英文摘要:

Large language models (LLMs) are now used worldwide, yet their multimodal understanding and reasoning often degrade outside Western, high-resource settings. We propose MMA-ASIA, a comprehensive framework to evaluate LLMs' cultural awareness with a focus on Asian contexts. MMA-ASIA centers on a human-curated, multilingual, and multimodally aligned multiple-choice benchmark covering 8 Asian countries and 10 languages, comprising 27,000 questions; over 79 percent require multi-step reasoning grounded in cultural context, moving beyond simple memorization. To our knowledge, this is the first dataset aligned at the input level across three modalities: text, image (visual question answering), and speech. This enables direct tests of cross-modal transfer. Building on this benchmark, we propose a five-dimensional evaluation protocol that measures: (i) cultural-awareness disparities across countries, (ii) cross-lingual consistency, (iii) cross-modal consistency, (iv) cultural knowledge generalization, and (v) grounding validity. To ensure rigorous assessment, a Cultural Awareness Grounding Validation Module detects "shortcut learning" by checking whether the requisite cultural knowledge supports correct answers. Finally, through comparative model analysis, attention tracing, and an innovative Vision-ablated Prefix Replay (VPR) method, we probe why models diverge across languages and modalities, offering actionable insights for building culturally reliable multimodal LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑