Evaluating Generative Models via One-Dimensional Code Distributions
WeChat AI, Tencent Inc.(腾讯微信AI);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院)
Zexi Jia, Pengcheng Luo, Yijia Zhong, Jinchao Zhang, Jie Zhou
大多数生成模型的评估依赖于特征分布度量标准,如FID,这些标准在连续识别特征上操作,这些特征被显式训练以对表现变化具有不变性,因此会丢弃对感知质量至关重要的线索。我们相反在离散视觉令牌的空间中评估模型,其中现代1D图像令牌化器能够紧凑地编码语义和感知信息,质量表现为可预测的令牌统计。我们引入了Codebook Histogram Distance (CHD),一种无需训练的令牌空间分布度量标准,以及Code Mixture Model Score (CMMS),一种从令牌序列的合成退化中学习的无参考质量度量标准。为了在广泛的分布转移下测试度量标准,我们进一步提出了VisForm,一个包含210万张图像、62种视觉形式和12种生成模型的基准,配有专家注释。在AGIQA、HPDv2/3和VisForm上,我们的基于令牌的度量标准在与人类判断的相关性上达到了最先进的水平。我们将发布所有代码和数据集以促进未来研究,代码已公开在https://github.com/zexiJia/1d-Distance。
英文摘要:
Most evaluations of generative models rely on feature-distribution metrics such as FID, which operate on continuous recognition features that are explicitly trained to be invariant to appearance variations, and thus discard cues critical for perceptual quality. We instead evaluate models in the space of discrete visual tokens, where modern 1D image tokenizers compactly encode both semantic and perceptual information and quality manifests as predictable token statistics. We introduce Codebook Histogram Distance (CHD), a training-free distribution metric in token space, and Code Mixture Model Score (CMMS), a no-reference quality metric learned from synthetic degradations of token sequences. To stress-test metrics under broad distribution shifts, we further propose VisForm, a benchmark of 210K images spanning 62 visual forms and 12 generative models with expert annotations. Across AGIQA, HPDv2/3, and VisForm, our token-based metrics achieve state-of-the-art correlation with human judgments. We will release all code and datasets to facilitate future research, with the code publicly available at https://github.com/zexiJia/1d-Distance.
MME-CoF-Pro: Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hints
Northeastern University(东北大学);The Chinese University of Hong Kong(香港中文大学);Westlake University(西湖大学);ByteDance Seed(字节跳动种子);Peking University(北京大学);NVIDIA(英伟达)
EvalGIM: A Library for Evaluating Generative Image Models
Mila, Quebec AI Institute(魁北克人工智能研究所(Mila));Univ. Grenoble Alpes(格勒诺布尔大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);Grenoble INP(格勒诺布尔理工学院);FAIR at Meta(元公司人工智能研究中心(FAIR));McGill University(麦吉尔大学)
Fréchet Power-Scenario Distance: A Metric for Evaluating Generative AI Models across Multiple Time-Scales in Smart Grids
Department of Electrical and Computer Engineering, Texas A&M University(电气与计算机工程系,德克萨斯农工大学);John A. Paulson School of Engineering and Applied Sciences, Harvard University(约翰·A·保罗森工程与应用科学学院,哈佛大学)
Scene Graph-Guided Generative AI Framework for Synthesizing and Evaluating Industrial Hazard Scenarios
Ph.D. Student, Dept. of Civil Eng., University of Texas at Arlington. E-mail;Assistant Professor, Dept. of Computer Sci. \& Eng., University of Texas at Arlington. E-mail;Assistant Professor, Dept. of Civil Eng., University of Texas at Arlington. E-mail
Evaluating the Clinical Impact of Generative Inpainting on Bone Age Estimation
Universidade Federal de São Paulo(巴西圣保罗联邦大学);Universidade de São Paulo(巴西圣保罗大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Dasa;Bunkerhill Health(Bunkerhill健康)
TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Models
University of British Columbia(不列颠哥伦比亚大学);Vector Institute for AI(Vector人工智能研究院);Toyota Technological Institute at Chicago(芝加哥丰田理工学院);Carleton College(卡尔顿学院);University of Pennsylvania(宾夕法尼亚大学)
Evaluating Compositional Scene Understanding in Multimodal Generative Models
Department of Psychology, University of California, Los Angeles(加州大学洛杉矶分校心理学系);Microsoft Research, NYC(微软研究院纽约分部);Air Force Research Laboratory(美国空军研究实验室);Department of Statistics, University of California, Los Angeles(加州大学洛杉矶分校统计学系)
T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models
UCAS(中国科学院大学);Tsinghua University(清华大学);Institute for AI(人工智能研究院);Tsinghua-Bosch Joint ML Center(清华-博世联合机器学习中心);THBI Lab(THBI实验室);BNRist Center(BNRist中心);RealAI