MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?
University of Chicago（芝加哥大学）；NUS（新加坡国立大学）；Stanford University（斯坦福大学）；UCSC（加州大学圣克鲁兹分校）；Duke University（杜克大学）；ESSEC（埃塞克高等商学院）；Illinois Tech（伊利诺伊理工学院）；UCSD（加州大学圣地亚哥分校）；Peking University（北京大学）；University of Queensland（昆士兰大学）；Stony Brook University（石溪大学）；UNC-Chapel Hill（北卡罗来纳大学教堂山分校）；USTC（中国科学技术大学）
While text-to-image models like DALLE-3 and Stable Diffusion are rapidly proliferating, they often encounter challenges such as hallucination, bias, and the production of unsafe, low-quality output. To effectively address these issues, it is crucial to align these models with desired behaviors based on feedback from a multimodal judge. Despite their significance, current multimodal judges frequently undergo inadequate evaluation of their capabilities and limitations, potentially leading to misalignment and unsafe fine-tuning outcomes. To address this issue, we introduce MJ-Bench, a novel benchmark which incorporates a comprehensive preference dataset to evaluate multimodal judges in providing feedback for image generation models across four key perspectives: alignment, safety, image quality, and bias. Specifically, we evaluate a large variety of multimodal judges including smaller-sized CLIP-based scoring models, open-source VLMs (e.g. LLaVA family), and close-source VLMs (e.g. GPT-4o, Claude 3) on each decomposed subcategory of our preference dataset. Experiments reveal that close-source VLMs generally provide better feedback, with GPT-4o outperforming other judges in average. Compared with open-source VLMs, smaller-sized scoring models can provide better feedback regarding text-image alignment and image quality, while VLMs provide more accurate feedback regarding safety and generation bias due to their stronger reasoning capabilities. Further studies in feedback scale reveal that VLM judges can generally provide more accurate and stable feedback in natural language (Likert-scale) than numerical scales. Notably, human evaluations on end-to-end fine-tuned models using separate feedback from these multimodal judges provide similar conclusions, further confirming the effectiveness of MJ-Bench. All data, code, models are available at https://huggingface.co/MJ-Bench.
Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis
University of Virginia（弗吉尼亚大学）；Columbia University（哥伦比亚大学）；Vanderbilt University（范德比大学）；Adobe Research（Adobe研究）；Dolby Laboratories（杜比实验室）；Cisco Research（思科研究）；University of Southern California（南加州大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of Oregon（俄勒冈大学）；Texas A&M University（德克萨斯大学）
Are We Really Benchmarking Forecasting Models? The Impact of Preprocessing on Time Series Performance
LIVIA, Department of Software and IT Engineering（LIVIA，软件与IT工程系）；École de technologie supérieure (ÉTS)（高等技术学院）；Center for Informatics (CIn)（信息学中心）；Universidade Federal de Pernambuco (UFPE)（伯南布哥联邦大学）
Institute for Artificial Intelligence, University of Stuttgart, Germany（人工智能研究所，斯图加特大学）；School of Informatics University of Edinburgh Edinburgh, UK（信息学院，爱丁堡大学）；University of Southampton, UK（南安普顿大学）；Stanford University（斯坦福大学）
Beyond IID: How General Are Tabular Foundation Models, Really?
Prior Labs；University of Freiburg（弗赖堡大学）；University of Mannheim（曼海姆大学）；INRIA Saclay（法国国家信息与自动化研究所萨克雷研究中心）；Technion（技术学院）；ELLIS Institute Tübingen（图宾根ELLIS研究所）；Zuse School ELIZA（Zuse学校ELIZA）
AGC-Bench: Measuring Artificial General Creativity
Pennsylvania State University（宾夕法尼亚州立大学）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；University of Amsterdam（阿姆斯特丹大学）；Amazon AGI（亚马逊AGI）；Dartmouth College（达特茅斯学院）
Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Artificial Intelligence, Shenzhen University（深圳大学人工智能学院）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Renmin University of China（中国人民大学）；National Yang Ming Chiao Tung University；Taipei Veterans General Hospital（台北荣民总医院）；School of Biomedical Engineering, Shenzhen University（深圳大学生物医学工程学院）
