Evaluating the alignment capabilities of large Vision-Language Models (VLMs) is essential for determining their effectiveness as helpful assistants. However, existing benchmarks primarily focus on basic abilities using nonverbal methods, such as yes-no and multiple-choice questions. In this paper, we address this gap by introducing AlignMMBench, which provides more nuanced evaluations of alignment capabilities and is the first benchmark specifically designed for Chinese visual contexts. This benchmark is meticulously curated from real-world scenarios and internet sources, encompassing thirteen specific tasks across three categories, and includes both single-turn and multi-turn dialogue scenarios. Incorporating a prompt rewrite strategy, AlignMMBench encompasses 1,054 images and 4,978 question-answer pairs. To facilitate the evaluation pipeline, we develop CritiqueVLM, a rule-calibrated evaluator that exceeds GPT-4's evaluation ability. Additionally, we measure the "alignment score", a quantitative metric designed to assess the robustness and stability of models across diverse prompts. Finally, we evaluate the performance of representative VLMs on AlignMMBench, offering insights into the capabilities and limitations of different VLM architectures. The evaluation code and data are available at https://github.com/THUDM/AlignMMBench.
Can MLLMs Understand the Deep Implication Behind Chinese Images?
Huazhong University of Science and Technology(华中科技大学);Shenzhen Institute of Advanced Technology, CAS(中国科学院深圳先进技术研究院);University of Science and Technology of China(中国科学技术大学);CDUT(成都理工大学);University of Memphis(孟菲斯大学);University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);SICAU(四川农业大学);University of Manchester(曼彻斯特大学);SWU(西南大学);UCAS(中国科学院大学);UMich(密歇根大学)
CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language
School of Informatics, Xiamen University, China(厦门大学信息学院);Key Lab of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian-Taiwan (XMU), Ministry of Culture and Tourism, China(福建省-台湾非物质文化遗产数字化保护与智能处理重点实验室(XMU),文化和旅游部,中国);National Language Resources Monitoring and Research Center for Education and Teaching Media, Xiamen University, China(教育与教学媒体语言资源监测与研究中心,厦门大学,中国)
Evaluating Chinese Ambiguity Understanding in Large Language Models
Graduate School of Information Science and Technology, The University of Tokyo(东京大学信息科学与技术研究生院);School of Software Engineering, South China University of Technology(华南理工大学软件学院)
CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patterns
Nanyang Technological University(南洋理工大学);Beijing University of Posts and Telecommunications(北京邮电大学);Huazhong University of Science and Technology(华中科技大学)
CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models
The Chinese University of Hong Kong(香港中文大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));LightSpeed Studios, Tencent Inc.(腾讯公司光速工作室);Sun Yat-Sen University(中山大学)
CHBench: A Chinese Dataset for Evaluating Health in Large Language Models
School of Artificial intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University(吉林大学知识驱动人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际中心)
Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratch
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学)
Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetry
College of Computer Science, Inner Mongolia University, China(内蒙古大学计算机学院,中国);National & Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, China(蒙古语智能信息处理技术国家与地方联合工程研究中心,中国);NLP²CT Lab, Department of Computer and Information Science, University of Macau, China(澳门大学计算机与信息科学学院NLP²CT实验室,中国)
CPsyExam: A Chinese Benchmark for Evaluating Psychology using Examinations
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Jilin University(吉林大学);University of Science and Technology of China(中国科学技术大学);Shenzhen University of Advanced Technology(深圳理工大学);University of Macau(澳门大学);Huazhong University of Science and Technology(华中科技大学)
Fùxì: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generation
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Lab of Artificial Intelligence for Education, East China Normal University(华东师范大学教育人工智能实验室);Shanghai Institute of Artificial Intelligence for Education, East China Normal University(华东师范大学上海教育人工智能研究院)
MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
East China Normal University(华东师范大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Hasso Plattner Institute(哈索·普拉特纳研究所);University of Potsdam(波茨坦大学);Cooperative Medianet Innovation Center, Shanghai Jiao Tong University(上海交通大学合作媒体网络创新中心)
Evaluating Reliability Asymmetries in Chinese Factual Search and AI Answers
Department of Electronics, Information and Bioengineering, Politecnico di Milano(电子、信息与生物工程系,米兰理工学院);University of Science and Technology of China(中国科学技术大学)