MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures
National University of Singapore(新加坡国立大学);Peking University(北京大学);Google DeepMind(谷歌DeepMind);Nanyang Technological University(南洋理工大学);Carnegie Mellon University(卡内基梅隆大学);University of Waterloo(滑铁卢大学)
Jinjie Ni, Yifan Song, Deepanway Ghosal, Bo Li, David Junhao Zhang, Xiang Yue, Fuzhao Xue, Zian Zheng, Kaichen Zhang, Mahir Shah, Kabir Jain, Yang You, Michael Shieh
Perceiving and generating diverse modalities are crucial for AI models to effectively learn from and engage with real-world signals, necessitating reliable evaluations for their development. We identify two major issues in current evaluations: (1) inconsistent standards, shaped by different communities with varying protocols and maturity levels; and (2) significant query, grading, and generalization biases. To address these, we introduce MixEval-X, the first any-to-any, real-world benchmark designed to optimize and standardize evaluations across diverse input and output modalities. We propose multi-modal benchmark mixture and adaptation-rectification pipelines to reconstruct real-world task distributions, ensuring evaluations generalize effectively to real-world use cases. Extensive meta-evaluations show our approach effectively aligns benchmark samples with real-world task distributions. Meanwhile, MixEval-X's model rankings correlate strongly with that of crowd-sourced real-world evaluations (up to 0.98) while being much more efficient. We provide comprehensive leaderboards to rerank existing models and organizations and offer insights to enhance understanding of multi-modal evaluations and inform future research.
Classifying Overlapping Gaussian Mixtures in High Dimensions: From Optimal Classifiers to Neural Nets
Raymond and Beverly Sackler School of Physics and Astronomy(雷蒙德和贝弗利·萨克勒物理与天文学院);Tel-Aviv University(特拉维夫大学);École Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院)
Addressing Corner Cases in Autonomous Driving: A World Model-based Approach with Mixture of Experts and LLMs
State Key Laboratory of Internet of Things for Smart City, University of Macau(物联网智能城市国家重点实验室,澳门大学);Department of Computer and Information Science, University of Macau(计算机与信息科学系,澳门大学);Department of Civil and Environmental Engineering, University of Macau(土木与环境工程系,澳门大学);Senseable City Lab, Massachusetts Institute of Technology(可感知城市实验室,麻省理工学院);Department of Civil and Environmental Engineering, University of Hawaii(土木与环境工程系,夏威夷大学)