Evaluating large language models at scale remains a practical bottleneck for many organizations. While existing evaluation frameworks work well for thousands of examples, they struggle when datasets grow to hundreds of thousands or millions of samples. This scale is common when assessing model behavior across diverse domains or conducting comprehensive regression testing. We present Spark-LLM-Eval, a distributed evaluation framework built natively on Apache Spark. The system treats evaluation as a data-parallel problem, partitioningexamplesacrossexecutorsandaggregatingresultswithproperstatistical accounting. Beyond raw throughput, we emphasize statistical rigor: every reported metric includes bootstrap confidence intervals, and model comparisons come with appropriate significance tests (paired t-tests, McNemar's test, or Wilcoxon signed-rank, depending on the metric type). The framework also addresses the cost problem inherent in LLM evaluation through content-addressable response caching backed by Delta Lake, which allows iterating on metric definitions without re-running inference. We describe the system architecture, the statistical methodology, and report benchmark results showing linear scaling with cluster size. The framework and all evaluation code are available as open source.
Sparks of Science: Hypothesis Generation Using Structured Paper Data
University of Oxford(牛津大学);Oak Ridge National Laboratory(橡树岭国家实验室);University College London(伦敦大学学院);University of Toronto(多伦多大学);Australian National University(澳大利亚国立大学);Modulos AG(Modulos AG公司);Stanford University(斯坦福大学)
The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic
Instituto Superior Técnico & INESC-ID, Universidade de Lisboa, Portugal(里斯本大学技术高级学院及INESC-ID研究所,里斯本大学,葡萄牙);Dept. of Computer Science and AI & DaSCI Institute, Universidad de Granada, Spain(计算机科学与人工智能系及DaSCI研究所,格拉纳达大学,西班牙)