arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.01020cs.DCcs.AI
收藏

用于 LLM 评估的 AI 基准与数据集

AI Benchmarks and Datasets for LLM Evaluation

Todor Ivanov, Valeri Penchev

AI总结:

本文介绍 AI Safety Bulgaria 下的基准收集与分类项目,通过整理 AI/LLM 评估基准,支撑全生命周期的定量评测、可信治理与合规实践。

摘要:

LLM 在预训练和微调两方面都需要大量计算资源,并且由于模型规模庞大而需要分布式计算能力。其复杂架构在从数据收集到部署和监控的整个 AI 生命周期中都带来挑战。应对可解释性、可纠正性、可诠释性和幻觉等关键 AI 系统挑战,需要系统化方法和严格的基准测试。为有效改进 AI 系统,我们必须通过定量评估精确识别系统性脆弱点,从而增强系统可信度。欧洲议会于 2024 年 3 月 13 日颁布《EU AI Act》,首次为 AI 系统的开发、部署和使用建立了覆盖全欧盟的综合性要求,这进一步凸显了 Z-Inspection 等工具和方法的重要性。该法案强调,需要用实用基准来丰富这一方法,以有效应对 AI 系统带来的技术挑战。为此,我们启动了一个隶属于 AI Safety Bulgaria 倡议的项目,旨在收集和分类 AI 基准。这将使从业者能够在 AI 系统整个生命周期中识别并使用这些基准。

英文摘要:

LLMs demand significant computational resources for both pre-training and fine-tuning, requiring distributed computing capabilities due to their large model sizes \cite{sastry2024computing}. Their complex architecture poses challenges throughout the entire AI lifecycle, from data collection to deployment and monitoring \cite{OECD_AIlifecycle}. Addressing critical AI system challenges, such as explainability, corrigibility, interpretability, and hallucination, necessitates a systematic methodology and rigorous benchmarking \cite{guldimann2024complai}. To effectively improve AI systems, we must precisely identify systemic vulnerabilities through quantitative evaluation, bolstering system trustworthiness. The enactment of the EU AI Act \cite{EUAIAct} by the European Parliament on March 13, 2024, establishing the first comprehensive EU-wide requirements for the development, deployment, and use of AI systems, further underscores the importance of tools and methodologies such as Z-Inspection. It highlights the need to enrich this methodology with practical benchmarks to effectively address the technical challenges posed by AI systems. To this end, we have launched a project that is part of the AI Safety Bulgaria initiatives \cite{AI_Safety_Bulgaria}, aimed at collecting and categorizing AI benchmarks. This will enable practitioners to identify and utilize these benchmarks throughout the AI system lifecycle.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑