HumaniBench: 一种以人为中心的大型多模态模型评估框架
HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation
AI总结：
本文提出HumaniBench框架，通过32000个专家验证的图像-问题对评估大型多模态模型在公平性、伦理、包容性等人类中心原则上的对齐情况，揭示了不同模型在伦理、推理和共情方面的权衡。
摘要：
尽管最近的大型多模态模型（LMMs）在视觉语言任务上取得了显著进展，但它们在公平性、伦理、包容性、共情和鲁棒性等以人为中心（HC）原则上的对齐往往被忽视。现有的LMM基准测试大多缺乏准确性考量。我们提出了HumaniBench，一个统一的框架，用于在现实、社会相关的视觉情境中表征HC对齐情况。它包含32000个经过专家验证的图像-问题对，这些对均映射到一个或多个HC原则通过显式指标。比较15种最先进的LMMs揭示了持续的权衡：专有系统在伦理、推理和共情方面领先，而开源模型在视觉基础和鲁棒性方面表现更优。所有模型在公平性和多语言包容性方面都存在持续的差距。链式思维提示和测试时扩展在几个HC维度上产生8至12%的提升。HumaniBench使能够对传统多模态基准未涵盖的对齐权衡进行细粒度分析。
英文摘要：
Although recent large multimodal models (LMMs) show impressive progress on vision language tasks, their alignment with human centered (HC) principles such as fairness, ethics, inclusivity, empathy, and robustness is often overlooked. Existing LMM benchmarks are largely accuracy-agnostic. We present HumaniBench, a unified framework for characterizing HC alignment across realistic, socially grounded visual contexts. It contains 32,000 expert-verified image-question pairs from real-world news imagery, each mapped to one or more HC principles through explicit metrics. Comparing 15 state of the art LMMs reveals consistent trade -offs: proprietary systems lead on ethics, reasoning, and empathy, while open-source models show superior visual grounding and resilience. All models show persistent gaps in fairness and multilingual inclusivity. Chain-of-thought prompting and test-time scaling yield 8to 12 % gains on several HC dimensions. HumaniBench enables fine-grained analysis of alignment trade-offs not captured by conventional multimodal benchmarks. https://vectorinstitute.github.io/humanibench/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
