本文介绍 AI Safety Bulgaria 下的基准收集与分类项目,通过整理 AI/LLM 评估基准,支撑全生命周期的定量评测、可信治理与合规实践。
摘要:
LLM 在预训练和微调两方面都需要大量计算资源,并且由于模型规模庞大而需要分布式计算能力。其复杂架构在从数据收集到部署和监控的整个 AI 生命周期中都带来挑战。应对可解释性、可纠正性、可诠释性和幻觉等关键 AI 系统挑战,需要系统化方法和严格的基准测试。为有效改进 AI 系统,我们必须通过定量评估精确识别系统性脆弱点,从而增强系统可信度。欧洲议会于 2024 年 3 月 13 日颁布《EU AI Act》,首次为 AI 系统的开发、部署和使用建立了覆盖全欧盟的综合性要求,这进一步凸显了 Z-Inspection 等工具和方法的重要性。该法案强调,需要用实用基准来丰富这一方法,以有效应对 AI 系统带来的技术挑战。为此,我们启动了一个隶属于 AI Safety Bulgaria 倡议的项目,旨在收集和分类 AI 基准。这将使从业者能够在 AI 系统整个生命周期中识别并使用这些基准。
英文摘要:
LLMs demand significant computational resources for both pre-training and fine-tuning, requiring distributed computing capabilities due to their large model sizes \cite{sastry2024computing}. Their complex architecture poses challenges throughout the entire AI lifecycle, from data collection to deployment and monitoring \cite{OECD_AIlifecycle}. Addressing critical AI system challenges, such as explainability, corrigibility, interpretability, and hallucination, necessitates a systematic methodology and rigorous benchmarking \cite{guldimann2024complai}. To effectively improve AI systems, we must precisely identify systemic vulnerabilities through quantitative evaluation, bolstering system trustworthiness. The enactment of the EU AI Act \cite{EUAIAct} by the European Parliament on March 13, 2024, establishing the first comprehensive EU-wide requirements for the development, deployment, and use of AI systems, further underscores the importance of tools and methodologies such as Z-Inspection. It highlights the need to enrich this methodology with practical benchmarks to effectively address the technical challenges posed by AI systems. To this end, we have launched a project that is part of the AI Safety Bulgaria initiatives \cite{AI_Safety_Bulgaria}, aimed at collecting and categorizing AI benchmarks. This will enable practitioners to identify and utilize these benchmarks throughout the AI system lifecycle.
Meituan(美团);Pennsylvania State University(宾夕法尼亚州立大学);Allen Institute for AI (AI2)(人工智能艾伦研究所);University of Washington(华盛顿大学);University at Buffalo(布法罗纽约州立大学)
OpenHospital: A Thing-in-itself Arena for Evolving and Benchmarking LLM-based Collective Intelligence
School of Software Technology, Zhejiang University(浙江大学软件学院);Laboratory for Statistical Monitoring and Intelligent Governance of Common Prosperity, Zhejiang Gongshang University(浙江工商大学共同富裕统计监测与智能治理实验室);Tongyi Lab, Alibaba Group(阿里集团通义实验室);Alibaba Cloud(阿里云)
LLMRec: Benchmarking Large Language Models on Recommendation Task
Alibaba Group(阿里巴巴集团);Ant Group(蚂蚁集团);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Peking University(北京大学);The Hong Kong University of Science and Technology(香港科技大学);University of Technology Sydney(悉尼科技大学);Yale University(耶鲁大学);University of Illinois Chicago(芝加哥伊利诺伊大学)
DABench-LLM: Standardized and In-Depth Benchmarking of Post-Moore Dataflow AI Accelerators for LLMs
Dept. of Computer Science Stevens Institute of Technology(计算机科学系 斯坦福理工学院);Dept. of Computer Science Binghamton University(计算机科学系 哈伯里顿大学);Engineering The Ohio State University(工程学 俄亥俄州立大学);Learning Division Argonne National Laboratory(学习部 阿贡国家实验室)
Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practices
Jožef Stefan Institute(乔塞夫·斯塔芬研究所);University of Ljubljana(卢布尔雅那大学);Faculty of Arts, University of Ljubljana(艺术学院,卢布尔雅那大学);Faculty of Computer and Information Science, University of Ljubljana(计算机与信息科学学院,卢布尔雅那大学);Institute of Contemporary History(现代史研究所)
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation
College of Computer Science and Artificial Intelligence, Fudan University(计算机科学与人工智能学院,复旦大学);Shanghai Key Lab of Intelligent Information Processing(上海智能信息处理重点实验室);Shanghai AI Laboratory(上海人工智能实验室);Northwestern University(西北大学)
University of Massachusetts, Amherst(马萨诸塞大学阿默斯特分校);Emory University(埃默里大学);University of Minnesota(明尼苏达大学);University of Massachusetts, Lowell(马萨诸塞大学洛厄尔分校);UMass Chan Medical School(UMass Chan医学学院)
Spanish and LLM Benchmarks: is MMLU Lost in Translation?
ETSI de Telecomunicación(电信高级工程师学院);Universidad Politécnica de Madrid(马德里理工大学);Facultad de Filosofía y Letras(哲学与文学学院);Universidad Autónoma de Madrid(马德里自治大学);SomosNLP