LLM系统预测性能的统计多指标评估与可视化
Statistical multi-metric evaluation and visualization of LLM system predictive performance
AI总结:
提出一个自动执行统计检验、聚合跨指标与数据集结果并可视化的框架,以支持LLM系统配置决策,并在CrossCodeEval基准上验证。
摘要:
对生成式或判别式大型语言模型(LLM)系统的评估通常是一个复杂的多维问题。通常,一组系统配置备选方案会在一个或多个基准数据集上进行评估,每个数据集带有一个或多个评估指标,这些指标在不同数据集间可能存在差异。我们通常希望以统计显著性度量来评估系统是否表现出差异:在给定数据集上根据单一指标、在数据集上跨指标聚合,或跨数据集。此类评估可支持决策,例如判定特定系统组件变更(如LLM选择或超参数值)是否显著提升当前系统配置的性能,或更一般地,判定固定系统配置集(如排行榜列表)是否根据关注指标表现出显著性能差异。我们提出一个框架实现,自动执行正确的统计检验,妥善聚合跨指标与数据集的统计结果(一项非平凡任务),并可视化结果。该框架在多语言代码生成基准CrossCodeEval上对若干最先进LLM进行了演示。
英文摘要:
The evaluation of generative or discriminative large language model (LLM)-based systems is often a complex multi-dimensional problem. Typically, a set of system configuration alternatives are evaluated on one or more benchmark datasets, each with one or more evaluation metrics, which may differ between datasets. We often want to evaluate -- with a statistical measure of significance -- whether systems perform differently either on a given dataset according to a single metric, on aggregate across metrics on a dataset, or across datasets. Such evaluations can be done to support decision-making, such as deciding whether a particular system component change (e.g., choice of LLM or hyperparameter values) significantly improves performance over the current system configuration, or, more generally, whether a fixed set of system configurations (e.g., a leaderboard list) have significantly different performances according to metrics of interest. We present a framework implementation that automatically performs the correct statistical tests, properly aggregates the statistical results across metrics and datasets (a nontrivial task), and can visualize the results. The framework is demonstrated on the multi-lingual code generation benchmark CrossCodeEval, for several state-of-the-art LLMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。