arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.13754cs.AIcs.LGcs.MM
收藏

MixEval-X:基于真实世界数据混合的任意到任意评估

MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures

National University of Singapore(新加坡国立大学);Peking University(北京大学);Google DeepMind(谷歌DeepMind);Nanyang Technological University(南洋理工大学);Carnegie Mellon University(卡内基梅隆大学);University of Waterloo(滑铁卢大学)

Jinjie Ni, Yifan Song, Deepanway Ghosal, Bo Li, David Junhao Zhang, Xiang Yue, Fuzhao Xue, Zian Zheng, Kaichen Zhang, Mahir Shah, Kabir Jain, Yang You, Michael Shieh

AI总结:

MixEval-X提出首个任意到任意真实世界多模态评估基准,通过多模态基准混合与适应-校正流程重建真实任务分布,实现与人工评估高达0.98的相关性,并显著提升评估效率。

摘要:

感知和生成多样化的模态对于AI模型有效地从真实世界信号中学习并与之交互至关重要,这需要可靠的评估来支持其发展。我们识别出当前评估中的两个主要问题:(1) 标准不一致,由不同社区以不同的协议和成熟度水平塑造而成;(2) 显著的查询、评分和泛化偏差。为了解决这些问题,我们引入了MixEval-X,这是第一个任意到任意、基于真实世界的基准,旨在优化和标准化跨多样化输入和输出模态的评估。我们提出了多模态基准混合和适应-校正流程来重建真实世界任务分布,确保评估能够有效泛化到真实世界使用场景。广泛的元评估表明,我们的方法有效地使基准样本与真实世界任务分布对齐。同时,MixEval-X的模型排名与基于众包的真实世界评估排名具有强相关性(最高达0.98),同时效率更高。我们提供了全面的排行榜来重新排名现有模型和组织,并提供见解以增强对多模态评估的理解并为未来研究提供参考。

英文摘要:

Perceiving and generating diverse modalities are crucial for AI models to effectively learn from and engage with real-world signals, necessitating reliable evaluations for their development. We identify two major issues in current evaluations: (1) inconsistent standards, shaped by different communities with varying protocols and maturity levels; and (2) significant query, grading, and generalization biases. To address these, we introduce MixEval-X, the first any-to-any, real-world benchmark designed to optimize and standardize evaluations across diverse input and output modalities. We propose multi-modal benchmark mixture and adaptation-rectification pipelines to reconstruct real-world task distributions, ensuring evaluations generalize effectively to real-world use cases. Extensive meta-evaluations show our approach effectively aligns benchmark samples with real-world task distributions. Meanwhile, MixEval-X's model rankings correlate strongly with that of crowd-sourced real-world evaluations (up to 0.98) while being much more efficient. We provide comprehensive leaderboards to rerank existing models and organizations and offer insights to enhance understanding of multi-modal evaluations and inform future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑