文本生成图像模型近年来进展显著,但该领域缺乏能准确反映模型性能的评估指标,尤其缺少可指导模型优化的细粒度指标。本文提出EvalAlign,一种兼具准确性、稳定性与细粒度特性的评估指标。我们的方法利用在海量数据上预训练的多模态大语言模型(Multimodal Large Language Models, MLLMs)的能力,围绕图像忠实度与文本-图像对齐两个核心维度构建评估协议。每个协议包含一组关联特定评分选项的详细细粒度指令,可对生成图像进行精准人工评分。我们对MLLM进行监督微调(Supervised Fine-Tune, SFT),使其与人类评估判断对齐,得到鲁棒的评估模型。我们在24个文本生成图像模型上的评估表明,EvalAlign不仅指标稳定性更优,且相比现有指标更贴合人类偏好,证实了其在模型评估中的有效性与实用性。
英文摘要:
The recent advancements in text-to-image generative models have been remarkable. Yet, the field suffers from a lack of evaluation metrics that accurately reflect the performance of these models, particularly lacking fine-grained metrics that can guide the optimization of the models. In this paper, we propose EvalAlign, a metric characterized by its accuracy, stability, and fine granularity. Our approach leverages the capabilities of Multimodal Large Language Models (MLLMs) pre-trained on extensive data. We develop evaluation protocols that focus on two key dimensions: image faithfulness and text-image alignment. Each protocol comprises a set of detailed, fine-grained instructions linked to specific scoring options, enabling precise manual scoring of the generated images. We supervised fine-tune (SFT) the MLLM to align with human evaluative judgments, resulting in a robust evaluation model. Our evaluation across 24 text-to-image generation models demonstrate that EvalAlign not only provides superior metric stability but also aligns more closely with human preferences than existing metrics, confirming its effectiveness and utility in model assessment.
University of Maryland, College Park(马里兰大学 College Park分校);University of Illinois Chicago(伊利诺伊大学芝加哥分校);University of Massachusetts Amherst(马萨诸塞大学阿姆赫斯特分校);Georgia Tech(佐治亚理工学院);Princeton University(普林斯顿大学)
Evaluating Vision-Language and Large Language Models for Automated Student Assessment in Indonesian Classrooms
Quantic School of Business and Technology(Quantic商业与技术学院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Indonesia University of Education(印度尼西亚教育大学);Monash University(莫纳什大学)
Disentangling Language and Culture for Evaluating Multilingual Large Language Models
Singapore Management University(新加坡管理学院);University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);Institute of Trustworthy Embodied AI, Fudan University(复旦大学可信具身人工智能研究院);DAMO Academy, Alibaba Group(阿里集团大模型实验室);Singapore University of Technology and Design(新加坡科技设计大学)
SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videos
School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);Center for Language and Information Research, Wuhan University(武汉大学语言与信息研究中心);Southwest Jiaotong University(西南交通大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学);The University of Manchester(曼彻斯特大学);Zhongnan Hospital of Wuhan University(武汉大学中南医院)
OralMLLM-Bench: Evaluating Cognitive Capabilities of Multimodal Large Language Models in Dental Practice
Beijing Stomatological Hospital(北京口腔医院);School of Stomatology, Capital Medical University(首都医科大学口腔医学院);University of Minnesota(明尼苏达大学);The Hong Kong Polytechnic University(香港理工大学)
InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews
Fudan University(复旦大学);Carnegie Mellon University(卡内基梅隆大学);The Chinese University of Hong Kong(香港中文大学);Renmin University of China(中国人民大学);Beijing University of Technology(北京工业大学);Boston University(波士顿大学);RhineAI(莱茵AI公司);SenseTime(商汤科技)
From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher
Department of Computer Science, King Khalid University(国王 Khalid 大学计算机科学系);Department of Computer Science and Artificial Intelligence, Umm Al-Qura University(乌姆·阿勒·库拉大学计算机科学与人工智能系);Imam Abdulrahman Bin Faisal University(伊玛目阿卜杜勒拉赫曼·本·法伊萨尔大学)
PAWBench: How Far Are We from Probabilistically Aligned World Modeling?
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Krea AI;Huggingface;Shanghai Innovation Institute(上海创新研究院);Tongyi Lab(通义实验室);The University of Hong Kong(香港大学)
VMBench: A Benchmark for Perception-Aligned Video Motion Generation
AMAP, Alibaba Group(阿里巴巴集团AMAP);CRISE, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所CRISE);School of Computer Science and Technology, University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院)
OpenLearnLM Benchmark: A Unified Framework for Evaluating Knowledge, Skill, and Attitude in Educational Large Language Models
Chosun University(chosun大学);Korea University(韩国大学);Ewha Womans University(成均馆大学);Korea Institute for Curriculum and Evaluation(韩国课程评价院);Seoul National University(首尔国立大学);Texas A&M University(德克萨斯农工大学);Indiana University Bloomington(印第安纳大学布卢明顿分校)
SAGI: Semantically Aligned and Uncertainty Guided AI Image Inpainting
Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki(阿尔伯塔大学电气与计算机工程系);Information Technologies Institute, CERTH(信息科技研究所)
SCOP: Evaluating the Comprehension Process of Large Language Models from a Cognitive View
Sichuan University(四川大学);Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education(教育部机器学习与工业智能工程研究中心);School of Statistics and Data Science, AAIS, Nankai University(南开大学统计与数据科学学院)
NativQA: Multilingual Culturally-Aligned Natural Query for LLMs
University of New Brunswick(新不伦瑞克大学);Qatar Computing Research Institute(卡塔尔计算研究中心);UPES(UPES大学);Carnegie Mellon University in Qatar(卡塔尔卡内基梅隆大学);Qatar University(卡塔尔大学)
InMind: Evaluating LLMs in Capturing and Applying Individual Human Reasoning Styles
Nankai University(南开大学);Shanghai Innovation Institute(上海创新研究院);Shanghai AI Laboratory(上海人工智能实验室);Fudan University(复旦大学);Johns Hopkins University(约翰霍普金斯大学);Wuhan University(武汉大学);University of Science and Technology of China(中国科学技术大学)
Can Agents Judge Systematic Reviews Like Humans? Evaluating SLRs with LLM-based Multi-Agent System
Department of Computer Science(计算机科学系);Information Technology University(信息技术大学);AI Center for Precision Health(精准健康人工智能中心);Weill Cornell Medicine–Qatar(韦尔·科恩医学中心–卡塔尔);Computer Science and Engineering Department(计算机科学与工程系)
WorldView-Bench: A Benchmark for Evaluating Global Cultural Perspectives in Large Language Models
Information Technology University Department of Computer Science(信息科技大学计算机科学系);Zayed University College of Interdisciplinary Studies(扎耶德大学跨学科研究学院);Computer Science and Engineering Department(计算机科学与工程系)