Assessing long-form responses generated by Vision-Language Models (VLMs) is challenging. It not only requires checking whether the VLM follows the given instruction but also verifying whether the text output is properly grounded on the given image. Inspired by the recent approach of evaluating LMs with LMs, in this work, we propose to evaluate VLMs with VLMs. For this purpose, we present a new feedback dataset called the Perception Collection, encompassing 15K customized score rubrics that users might care about during assessment. Using the Perception Collection, we train Prometheus-Vision, the first open-source VLM evaluator model that can understand the user-defined score criteria during evaluation. Prometheus-Vision shows the highest Pearson correlation with human evaluators and GPT-4V among open-source models, showing its effectiveness for transparent and accessible evaluation of VLMs. We open-source our code, dataset, and model at https://github.com/kaistAI/prometheus-vision
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
KAIST(韩国科学技术院);LG AI Research(LG人工智能研究院);Carnegie Mellon University(卡内基梅隆大学);MIT(麻省理工学院);Allen Institute for AI(艾伦人工智能研究所);University of Illinois Chicago(芝加哥伊利诺伊大学)
Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis
University of Virginia(弗吉尼亚大学);Columbia University(哥伦比亚大学);Vanderbilt University(范德比大学);Adobe Research(Adobe研究);Dolby Laboratories(杜比实验室);Cisco Research(思科研究);University of Southern California(南加州大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);University of Oregon(俄勒冈大学);Texas A&M University(德克萨斯大学)
Judge Before Answer: Can MLLM Discern the False Premise in Question?
School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院);Inner Mongolia Research Institute, Shanghai Jiao Tong University(上海交通大学内蒙古研究院)
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models
Carnegie Mellon University(卡内基梅隆大学);KAIST(韩国科学技术院);LG AI Research(LG人工智能研究院);Cornell University(康奈尔大学);MIT(麻省理工学院);Yonsei University(延世大学);University of Washington(华盛顿大学);University of Illinois Chicago(芝加哥大学伊利诺伊分校)
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
The University of Hong Kong(香港大学);Nanjing University(南京大学);University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);Fudan University(复旦大学)
HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessment
The State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Zhejiang University(浙江大学);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院);Pretraining Team, xAI(预训练团队,xAI);Hong Kong University of Science and Technology (GuangZhou)(香港科技大学(广州))
IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University(清华大学对话人工智能(CoAI)小组);Zhipu AI(智谱AI);University of Electronic Science and Technology of China(电子科技大学);The Knowledge Engineering Group (KEG), Tsinghua University(清华大学知识工程小组(KEG))
On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China(中国科学院软件研究所信息处理实验室,北京,中国);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国);Xiaohongshu Inc(小红书公司)
EmplifAI: a Fine-grained Dataset for Japanese Empathetic Medical Dialogues in 28 Emotion Labels
Kyoto Institute of Technology, Japan(京都技术大学);National Institute of Information and Communications Technology (NICT), Japan(信息通信技术国家研究所);Kyoto University, Japan(京都大学);Nara Institute of Science and Technology, Japan(奈良科学技术大学)
From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University(数据科学上海重点实验室,计算机科学与人工智能学院,复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation
Digital Twin Institute, Eastern Institute of Technology, Ningbo, China(数字孪生研究所,东部技术研究所,宁波,中国);Saarland University(萨尔兰州大学);Department of Computing, The Hong Kong Polytechnic University(计算系,香港理工大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search
Nanyang Technological University(南洋理工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Wuhan University(武汉大学);National University of Singapore(新加坡国立大学);University of New South Wales(新南威尔士大学);MiroMind;The Chinese University of Hong Kong(香港中文大学)
RPC-Bench: A Fine-grained Benchmark for Research Paper Comprehension
Xinjiang University(新疆大学);Renmin University of China(中国人民大学);Anhui University(安徽大学);Sun Yat-sen University(中山大学);Tsinghua University(清华大学);University of Southampton(南安普顿大学)
P2P: Automated Paper-to-Poster Generation and Fine-Grained Benchmark
ByteDance(字节跳动);M-A-P;College of Computer Science, Beijing Information Science and Technology University(北京信息科技大学计算机学院);Shenzhen Intelligent Strong Technology Co.,Ltd.(深圳智强科技有限公司)
CDJUR-BR -- A Golden Collection of Legal Document from Brazilian Justice with Fine-Grained Named Entities
Universidade de Fortaleza(福塔莱萨大学);Instituto Federal de Educação, Ciência e Tecnologia do Ceará(塞阿拉联邦教育、科学与技术学院);Tribunal de Justiça do Estado do Ceará(塞阿拉州法院)