arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.28769cs.DCcs.CLcs.LG
收藏

Spark-LLM-Eval: 一个用于统计严谨大语言模型评估的分布式框架

Spark-LLM-Eval: A Distributed Framework for Statistically Rigorous Large Language Model Evaluation

Subhadip Mitra

AI总结:

本文提出Spark-LLM-Eval,一个基于Apache Spark的分布式评估框架,通过数据并行处理实现大规模LLM评估,强调统计严谨性,提供置信区间和显著性检验,并利用Delta Lake实现响应缓存以降低评估成本。

摘要:

大规模评估大语言模型仍然是许多组织的实际瓶颈。尽管现有评估框架在处理数千个示例时表现良好,但当数据集增长到数万或数百万样本时,它们会遇到困难。这种规模在评估模型行为跨不同领域或进行全面回归测试时很常见。我们提出了Spark-LLM-Eval,一个原生基于Apache Spark的分布式评估框架。该系统将评估视为数据并行问题,将示例分配到执行程序中,并通过适当的统计会计进行结果汇总。除了原始吞吐量外,我们强调统计严谨性:每个报告的指标都包含置信区间,模型比较都配有适当的显著性检验(配对t检验、McNemar检验或Wilcoxon符号秩检验,根据指标类型而定)。该框架还通过内容寻址响应缓存(由Delta Lake支持)解决了LLM评估固有的成本问题,这允许在不重新运行推理的情况下迭代指标定义。我们描述了系统架构、统计方法,并报告了基准结果,显示与集群规模成线性增长。该框架和所有评估代码均作为开源提供。

英文摘要:

Evaluating large language models at scale remains a practical bottleneck for many organizations. While existing evaluation frameworks work well for thousands of examples, they struggle when datasets grow to hundreds of thousands or millions of samples. This scale is common when assessing model behavior across diverse domains or conducting comprehensive regression testing. We present Spark-LLM-Eval, a distributed evaluation framework built natively on Apache Spark. The system treats evaluation as a data-parallel problem, partitioningexamplesacrossexecutorsandaggregatingresultswithproperstatistical accounting. Beyond raw throughput, we emphasize statistical rigor: every reported metric includes bootstrap confidence intervals, and model comparisons come with appropriate significance tests (paired t-tests, McNemar's test, or Wilcoxon signed-rank, depending on the metric type). The framework also addresses the cost problem inherent in LLM evaluation through content-addressable response caching backed by Delta Lake, which allows iterating on metric definitions without re-running inference. We describe the system architecture, the statistical methodology, and report benchmark results showing linear scaling with cluster size. The framework and all evaluation code are available as open source.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑